[GitHub] metron pull request #1122: METRON-1683: Fix the download progress bar in PCA...
Github user sardell closed the pull request at: https://github.com/apache/metron/pull/1122 ---
[jira] [Commented] (METRON-1683) PCAP UI - Fix the download progress bar
[ https://issues.apache.org/jira/browse/METRON-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16572870#comment-16572870 ] ASF GitHub Bot commented on METRON-1683: Github user sardell closed the pull request at: https://github.com/apache/metron/pull/1122 > PCAP UI - Fix the download progress bar > --- > > Key: METRON-1683 > URL: https://issues.apache.org/jira/browse/METRON-1683 > Project: Metron > Issue Type: Sub-task >Reporter: Shane Ardell >Priority: Major > > The download progress bar currently does not appear when a PCAP search is > submitted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1134: METRON-1696: Create the HDFS directory for pcap sequence...
Github user MohanDV commented on the issue: https://github.com/apache/metron/pull/1134 Thanks for pointing this out @merrimanr, Yes this will overwrite the the #1019 . If the feature branch is not getting to master sooner then we can keep this change and rebase the feature branch else we can discard this. @mmiklavc metron:metron should allow pcap topology to write to hdfs directory. I started the topology as metron user. ---
[jira] [Commented] (METRON-1696) Pcap parser fails to write pacap sequence file to hdfs on kerberized cluster
[
https://issues.apache.org/jira/browse/METRON-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16572912#comment-16572912
]
ASF GitHub Bot commented on METRON-1696:
Github user MohanDV commented on the issue:
https://github.com/apache/metron/pull/1134
Thanks for pointing this out @merrimanr, Yes this will overwrite the the
#1019 . If the feature branch is not getting to master sooner then we can keep
this change and rebase the feature branch else we can discard this. @mmiklavc
metron:metron should allow pcap topology to write to hdfs directory. I started
the topology as metron user.
> Pcap parser fails to write pacap sequence file to hdfs on kerberized cluster
> -
>
> Key: METRON-1696
> URL: https://issues.apache.org/jira/browse/METRON-1696
> Project: Metron
> Issue Type: Bug
>Reporter: Mohan
>Assignee: Mohan
>Priority: Major
>
> pcap parser fails to write the pcap sequence files to hdfs directory due to
> insufficient privileges to hdfs folder for 'metron' user
> {code:java}
> 2018-07-25 10:15:50.035 o.a.m.s.p.HDFSWriterCallback
> Thread-9-kafkaSpout-executor[3 3] [ERROR] Permission denied: user=metron,
> access=WRITE,
> inode="/apps/metron/pcap/pcap_pcap_1532513746365022000_0_pcap-20-1532414055":hdfs:hdfs:drwxr-xr-x
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:353)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:325)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:246)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1950)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1934)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1917)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2767)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2702)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2586)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:736)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:409)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1134: METRON-1696: Create the HDFS directory for pcap sequence...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1134 > metron:metron did not allow pcap topology to write as the Storm user. Yup @mmiklavc .. on a non-kerberized cluster, I faced the same issue. ---
[jira] [Commented] (METRON-1696) Pcap parser fails to write pacap sequence file to hdfs on kerberized cluster
[
https://issues.apache.org/jira/browse/METRON-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16572930#comment-16572930
]
ASF GitHub Bot commented on METRON-1696:
Github user anandsubbu commented on the issue:
https://github.com/apache/metron/pull/1134
> metron:metron did not allow pcap topology to write as the Storm user.
Yup @mmiklavc .. on a non-kerberized cluster, I faced the same issue.
> Pcap parser fails to write pacap sequence file to hdfs on kerberized cluster
> -
>
> Key: METRON-1696
> URL: https://issues.apache.org/jira/browse/METRON-1696
> Project: Metron
> Issue Type: Bug
>Reporter: Mohan
>Assignee: Mohan
>Priority: Major
>
> pcap parser fails to write the pcap sequence files to hdfs directory due to
> insufficient privileges to hdfs folder for 'metron' user
> {code:java}
> 2018-07-25 10:15:50.035 o.a.m.s.p.HDFSWriterCallback
> Thread-9-kafkaSpout-executor[3 3] [ERROR] Permission denied: user=metron,
> access=WRITE,
> inode="/apps/metron/pcap/pcap_pcap_1532513746365022000_0_pcap-20-1532414055":hdfs:hdfs:drwxr-xr-x
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:353)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:325)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:246)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1950)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1934)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1917)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2767)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2702)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2586)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:736)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:409)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1142: METRON-1712: PCAP UI Input validation
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1142 Hmm, the port errors seem to have been a browser caching issue on the page. I looked at again and it validate ports properly. Sorry for the false alarm. ---
[jira] [Commented] (METRON-1723) PCAP UI - Unable to select/copy from packets details in PCAP query panel
[ https://issues.apache.org/jira/browse/METRON-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573183#comment-16573183 ] ASF GitHub Bot commented on METRON-1723: Github user sardell commented on the issue: https://github.com/apache/metron/pull/1139 @justinleet Those merge conflicts are now resolved. > PCAP UI - Unable to select/copy from packets details in PCAP query panel > > > Key: METRON-1723 > URL: https://issues.apache.org/jira/browse/METRON-1723 > Project: Metron > Issue Type: Sub-task >Reporter: Shane Ardell >Priority: Major > Attachments: expand-collapse-details-PCAP-query.mp4 > > > Upon query completion, a user can click on a row and drill down (up to 2 > levels) for a given packet. > A SOC analyst might find it useful to select/copy specific information from > the 2nd level granular details (E.g. MAC address, IP address, timestamps > etc.) to be used subsequently for analysis/investigation. > Currently, selecting or clicking on nested content fires an event that > expands/collapses the information. See the attached video to view the > behavior. > It would be better if target for the collapse/expand event only applied to > the top of the details section. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1139: METRON-1723: Unable to select/copy from packets details ...
Github user sardell commented on the issue: https://github.com/apache/metron/pull/1139 @justinleet Those merge conflicts are now resolved. ---
[jira] [Commented] (METRON-1712) PCAP UI - Input validation
[ https://issues.apache.org/jira/browse/METRON-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573180#comment-16573180 ] ASF GitHub Bot commented on METRON-1712: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1142 Hmm, the port errors seem to have been a browser caching issue on the page. I looked at again and it validate ports properly. Sorry for the false alarm. > PCAP UI - Input validation > -- > > Key: METRON-1712 > URL: https://issues.apache.org/jira/browse/METRON-1712 > Project: Metron > Issue Type: Sub-task >Reporter: Shane Ardell >Priority: Major > > The following fields need input validation: > IP Source Address: should accept valid IPv4 address formats (v6 is not yet > supported) > IP Source Port: should accept integers from 0 to 65535 > IP Source Address: should accept valid IPv4 address formats (v6 is not yet > supported) > IP Dest Port: should accept integers from 0 to 65535 > Leave unvalidated: > Protocol: number of valid protocols is around 150 and it could change in the > future, validation possible makes more sense on the backend or by a separated > endpoint > Free text filtering: binary regular expression > Broken inputs should be recognized by the UI and not submitted, with > user-facing feedback. The REST API should also fail on queries outside > reasonable parameters. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1153: METRON-1725: Add ability to specify YARN queue fo...
GitHub user merrimanr opened a pull request: https://github.com/apache/metron/pull/1153 METRON-1725: Add ability to specify YARN queue for pcap jobs ## Contributor Comments This PR exposes a configuration option in the Pcap CLI and REST app for the YARN queue a pcap job will be submitted to. This configuration is passed on to the `mapreduce.job.queuename` property in the Hadoop config. A YARN queue is set in the Pcap CLI with an additional command line option (`-yq`) and is set in the REST app as a spring property (`pcap.yarn.queue`). ### Testing This has been tested in full dev: 1. Create a YARN queue named `pcap` using the YARN Queue Manager 2. Ambari should prompt you, but be sure to restart the Resource Manager 3. Submit a job with the Pcap CLI and add the YARN queue option: `-yq pcap` 4. While the job is running navigate to the Resource Manager UI and verify the job is running in the `pcap` queue 5. Try to submit a job to a queue that doesn't exist. You should get a clear error stating the queue is missing 6. Go to Ambari > Services > Metron > REST and set the `Metron Spring options` property to `--pcap.yarn.queue=pcap`. 7. Ambari will prompt you to restart REST 8. Submit a job through the REST app and verify the job is running in the `pcap` queue 9. Submitting jobs in the Pcap CLI without the `yq` option and submitting jobs in REST without setting the `pcap.yarn.queue` should submit to the default queue. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/merrimanr/incubator-metron METRON-1725 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1153.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1153 commit bf17024766edc9a48d6c915001775315ba6e3f19 Author: merrimanr Date: 2018-08-06T22:03:14Z initial commit commit c7d9108fcfe5f3ab5a08a55e56e0272cc1dac4a2 Author: merrimanr Date: 2018-08-08T12:37:49Z added tests and documentation ---
[jira] [Commented] (METRON-1725) Add ability to specify YARN queue for pcap jobs
[ https://issues.apache.org/jira/browse/METRON-1725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573187#comment-16573187 ] ASF GitHub Bot commented on METRON-1725: GitHub user merrimanr opened a pull request: https://github.com/apache/metron/pull/1153 METRON-1725: Add ability to specify YARN queue for pcap jobs ## Contributor Comments This PR exposes a configuration option in the Pcap CLI and REST app for the YARN queue a pcap job will be submitted to. This configuration is passed on to the `mapreduce.job.queuename` property in the Hadoop config. A YARN queue is set in the Pcap CLI with an additional command line option (`-yq`) and is set in the REST app as a spring property (`pcap.yarn.queue`). ### Testing This has been tested in full dev: 1. Create a YARN queue named `pcap` using the YARN Queue Manager 2. Ambari should prompt you, but be sure to restart the Resource Manager 3. Submit a job with the Pcap CLI and add the YARN queue option: `-yq pcap` 4. While the job is running navigate to the Resource Manager UI and verify the job is running in the `pcap` queue 5. Try to submit a job to a queue that doesn't exist. You should get a clear error stating the queue is missing 6. Go to Ambari > Services > Metron > REST and set the `Metron Spring options` property to `--pcap.yarn.queue=pcap`. 7. Ambari will prompt you to restart REST 8. Submit a job through the REST app and verify the job is running in the `pcap` queue 9. Submitting jobs in the Pcap CLI without the `yq` option and submitting jobs in REST without setting the `pcap.yarn.queue` should submit to the default queue. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/merrimanr/incubator-metron METRON-1725 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1153.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1153 commit bf17024766edc9a48d6c91500177
[jira] [Assigned] (METRON-1556) Create YARN service
[ https://issues.apache.org/jira/browse/METRON-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman reassigned METRON-1556: - > Create YARN service > --- > > Key: METRON-1556 > URL: https://issues.apache.org/jira/browse/METRON-1556 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > We need a YARN Service in REST for interacting with the YARN REST api. The > service should at least: > * Return a description of a job including status, start time, end time, and > any other relevant information given a job id > * Stop a job given a job id > The > [StormStatusService|https://github.com/apache/metron/blob/master/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormStatusServiceImpl.java] > is similar in that it also proxies to a REST API. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Deleted] (METRON-1556) Create YARN service
[ https://issues.apache.org/jira/browse/METRON-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman deleted METRON-1556: -- > Create YARN service > --- > > Key: METRON-1556 > URL: https://issues.apache.org/jira/browse/METRON-1556 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > We need a YARN Service in REST for interacting with the YARN REST api. The > service should at least: > * Return a description of a job including status, start time, end time, and > any other relevant information given a job id > * Stop a job given a job id > The > [StormStatusService|https://github.com/apache/metron/blob/master/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormStatusServiceImpl.java] > is similar in that it also proxies to a REST API. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1638) Retrieve Pcap results in pdml format
[ https://issues.apache.org/jira/browse/METRON-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman reassigned METRON-1638: - Assignee: Ryan Merriman > Retrieve Pcap results in pdml format > > > Key: METRON-1638 > URL: https://issues.apache.org/jira/browse/METRON-1638 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > There should be a REST endpoint that allows a user to retrieve pcap page > results in pdml format. Assuming tshark is installed, there should be a "GET > /api/v1/pcap/pdml//" endpoint that will return pcap > results for the given page in pdml format > ([https://wiki.wireshark.org/PDML]), converted to json for easier consumption > by a UI. This endpoint will call out to the tskark utility for the raw to > pdml conversion. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1562) Enable Kerberos in REST for YARN and MR jobs
[ https://issues.apache.org/jira/browse/METRON-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman reassigned METRON-1562: - Assignee: Ryan Merriman > Enable Kerberos in REST for YARN and MR jobs > > > Key: METRON-1562 > URL: https://issues.apache.org/jira/browse/METRON-1562 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > We need to enable Kerberos support in REST for YARN and MR interactions. > This will include: > * Kerberos authentication for YARN REST api > * Kerberos authentication for submitting MR jobs > * Kerberos authentication for accessing query results (this is likely > already done but should be validated) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1560) Update MPack to support Pcap panel
[ https://issues.apache.org/jira/browse/METRON-1560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman reassigned METRON-1560: - Assignee: Ryan Merriman > Update MPack to support Pcap panel > -- > > Key: METRON-1560 > URL: https://issues.apache.org/jira/browse/METRON-1560 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > The MPack will need to be adjusted to support the Pcap panel. This includes: > * prompting a user during the MPack installation process to agree to tshark > being installed > * installing tshark on the correct node(s) > * adding pcap related properties to the MPack so they can be managed in > Ambari -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1685) Retrieve Pcap results in raw binary format
[ https://issues.apache.org/jira/browse/METRON-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman reassigned METRON-1685: - Assignee: Ryan Merriman > Retrieve Pcap results in raw binary format > -- > > Key: METRON-1685 > URL: https://issues.apache.org/jira/browse/METRON-1685 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > There should be a REST endpoint that allows a user to download pcap page > results in binary format. There should be a "GET > /api/v1/pcap/raw//" endpoint that will return a pcap file > containing pcaps for that page. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1700) Create REST endpoint to get job configuration
[ https://issues.apache.org/jira/browse/METRON-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman reassigned METRON-1700: - Assignee: Ryan Merriman > Create REST endpoint to get job configuration > - > > Key: METRON-1700 > URL: https://issues.apache.org/jira/browse/METRON-1700 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > We need a REST endpoint that will allow us to retrieve the job configuration > properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1721) New default input path is wrong in pcap CLI
[ https://issues.apache.org/jira/browse/METRON-1721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman reassigned METRON-1721: - Assignee: Ryan Merriman > New default input path is wrong in pcap CLI > --- > > Key: METRON-1721 > URL: https://issues.apache.org/jira/browse/METRON-1721 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > We needed a separate HDFS directory to store pcap files so the default was > changed from /apps/metron/pcap to /apps/metron/pcap/input. The CLI should be > updated to reflect this change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1691) REST should limit the number of Pcap jobs a user can submit
[ https://issues.apache.org/jira/browse/METRON-1691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman reassigned METRON-1691: - Assignee: Ryan Merriman > REST should limit the number of Pcap jobs a user can submit > --- > > Key: METRON-1691 > URL: https://issues.apache.org/jira/browse/METRON-1691 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > The REST app should keep users from submitting an unbounded number of > simultaneous jobs and overwhelming the cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Deleted] (METRON-1559) Create Pcap Service
[ https://issues.apache.org/jira/browse/METRON-1559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman deleted METRON-1559: -- > Create Pcap Service > --- > > Key: METRON-1559 > URL: https://issues.apache.org/jira/browse/METRON-1559 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Priority: Major > > We need a service that exposes the various Pcap endpoints. These include: > * GET /api/v1/pcap/metadata?basePath - This endpoint will return metadata of > pcap data stored in HDFS. This would include pcap size, date ranges (how far > back can I go), etc. It would accept an optional HDFS basePath parameter for > cases where pcap data is stored in multiple places and/or different from the > default location. > * POST /api/v1/pcap/fixed - This endpoint would accept a fixed pcap request, > submit a pcap job, and return a job id. The request would be an object > containing the options documented here for the fixed filter: > [https://github.com/apache/metron/tree/master/metron-platform/metron-pcap-backend#query-filter-utility]. > A job will be associated with a user that submits it. An exception will be > returned for violating constraints like too many queries submitted, query > parameters out of limits, etc. A record of the user and job id will be > persisted to a data store so a list of a user's jobs can later be retrieved. > * POST /api/v1/pcap/query - This endpoint would accept a query pcap request, > submit a pcap job, and return a job id. The request would be an object > containing the options documented here for the query filter: > [https://github.com/apache/metron/tree/master/metron-platform/metron-pcap-backend#query-filter-utility]. > A job will be associated with a user that submits it. An exception will be > returned for violating constraints like too many queries submitted, query > parameters out of limits, etc. A record of the user and job id will be > persisted to a data store so a list of a user's jobs can later be retrieved. > * GET /api/v1/pcap/status/ - This endpoint will return the YARN > status of a running/completed job. > * GET /api/v1/pcap/stop/ - This endpoint would kill a running pcap > job. If the job has already completed this is a noop. > * GET /api/v1/pcap/list - This endpoint will list a user's submitted pcap > queries. Items in the list would contain job id, status (is it finished?), > start/end time, and number of pages. > * GET /api/v1/pcap/pdml// - This endpoint will return > pcap results for the given page in pdml format > ([https://wiki.wireshark.org/PDML]). Are there other formats we want to > support? > * GET /api/v1/pcap/raw// - This endpoint will allow a > user to download raw pcap results for the given page. > * DELETE /api/v1/pcap/ - This endpoint will delete pcap query results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1674) Create REST endpoint for job status abstraction
[ https://issues.apache.org/jira/browse/METRON-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman reassigned METRON-1674: - Assignee: Ryan Merriman > Create REST endpoint for job status abstraction > --- > > Key: METRON-1674 > URL: https://issues.apache.org/jira/browse/METRON-1674 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > We need a REST endpoint that will enable us to get the status of a running > job. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1142: METRON-1712: PCAP UI Input validation
Github user tiborm commented on the issue: https://github.com/apache/metron/pull/1142 Thanks, Justin! I replaced the regexp for IP validation and added new tests to cover that functionality. Nice catch! I was unable to reproduce the port validation issue you reported. However, I replaced the regexp and double checked the tests. They seem working fine. ---
[jira] [Commented] (METRON-1712) PCAP UI - Input validation
[ https://issues.apache.org/jira/browse/METRON-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573210#comment-16573210 ] ASF GitHub Bot commented on METRON-1712: Github user tiborm commented on the issue: https://github.com/apache/metron/pull/1142 Thanks, Justin! I replaced the regexp for IP validation and added new tests to cover that functionality. Nice catch! I was unable to reproduce the port validation issue you reported. However, I replaced the regexp and double checked the tests. They seem working fine. > PCAP UI - Input validation > -- > > Key: METRON-1712 > URL: https://issues.apache.org/jira/browse/METRON-1712 > Project: Metron > Issue Type: Sub-task >Reporter: Shane Ardell >Priority: Major > > The following fields need input validation: > IP Source Address: should accept valid IPv4 address formats (v6 is not yet > supported) > IP Source Port: should accept integers from 0 to 65535 > IP Source Address: should accept valid IPv4 address formats (v6 is not yet > supported) > IP Dest Port: should accept integers from 0 to 65535 > Leave unvalidated: > Protocol: number of valid protocols is around 150 and it could change in the > future, validation possible makes more sense on the backend or by a separated > endpoint > Free text filtering: binary regular expression > Broken inputs should be recognized by the UI and not submitted, with > user-facing feedback. The REST API should also fail on queries outside > reasonable parameters. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1142: METRON-1712: PCAP UI Input validation
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/1142#discussion_r208575081
--- Diff:
metron-interface/metron-alerts/src/app/pcap/pcap-filters/pcap-filters.component.spec.ts
---
@@ -331,13 +337,60 @@ describe('PcapFiltersComponent', () => {
validValues.forEach((value) => {
const els = getFieldWithSubmit('ip-dest-port');
+expect(isFieldInvalid(els.field)).toBe(false, 'the field should be
valid without ' + value);
+expect(isSubmitDisabled(els.submit)).toBe(false, 'the submit
button should be enabled without ' + value);
+
+setFieldValue(els.field, value);
+
expect(isFieldInvalid(els.field)).toBe(false, 'the field should be
valid with ' + value);
expect(isSubmitDisabled(els.submit)).toBe(false, 'the submit
button should be enabled with ' + value);
+tearDown(els.field);
+ });
+});
+
+
+it('should disable the form if the ip source field is invalid', () => {
+ const invalidValues = [
+'tst',
+0o0,
+0,
+'111.111.111',
+'222.222.222.222.222',
+'333.333.333.333',
+ ];
+
+ invalidValues.forEach((value) => {
+const els = getFieldWithSubmit('ip-src-addr');
+expect(isFieldInvalid(els.field)).toBe(false, 'the field should be
valid without ' + value);
+expect(isSubmitDisabled(els.submit)).toBe(false, 'the submit
button should be enabled without ' + value);
+
setFieldValue(els.field, value);
-expect(isFieldInvalid(els.field)).toBe(false, 'the field should be
valid with ' + value);
+
+expect(isFieldInvalid(els.field)).toBe(true, 'the field should be
invalid with ' + value);
+expect(isSubmitDisabled(els.submit)).toBe(true, 'the submit button
should be disabled with ' + value);
+tearDown(els.field);
+ });
+});
+
+it('should keep the form enabled if the ip source field is valid', ()
=> {
+ const invalidValues = [
--- End diff --
Looks like this variable is misnamed, can you rename this (and it's usage
in at 382)to `validValues`?
---
[jira] [Commented] (METRON-1712) PCAP UI - Input validation
[
https://issues.apache.org/jira/browse/METRON-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573218#comment-16573218
]
ASF GitHub Bot commented on METRON-1712:
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/1142#discussion_r208575081
--- Diff:
metron-interface/metron-alerts/src/app/pcap/pcap-filters/pcap-filters.component.spec.ts
---
@@ -331,13 +337,60 @@ describe('PcapFiltersComponent', () => {
validValues.forEach((value) => {
const els = getFieldWithSubmit('ip-dest-port');
+expect(isFieldInvalid(els.field)).toBe(false, 'the field should be
valid without ' + value);
+expect(isSubmitDisabled(els.submit)).toBe(false, 'the submit
button should be enabled without ' + value);
+
+setFieldValue(els.field, value);
+
expect(isFieldInvalid(els.field)).toBe(false, 'the field should be
valid with ' + value);
expect(isSubmitDisabled(els.submit)).toBe(false, 'the submit
button should be enabled with ' + value);
+tearDown(els.field);
+ });
+});
+
+
+it('should disable the form if the ip source field is invalid', () => {
+ const invalidValues = [
+'tst',
+0o0,
+0,
+'111.111.111',
+'222.222.222.222.222',
+'333.333.333.333',
+ ];
+
+ invalidValues.forEach((value) => {
+const els = getFieldWithSubmit('ip-src-addr');
+expect(isFieldInvalid(els.field)).toBe(false, 'the field should be
valid without ' + value);
+expect(isSubmitDisabled(els.submit)).toBe(false, 'the submit
button should be enabled without ' + value);
+
setFieldValue(els.field, value);
-expect(isFieldInvalid(els.field)).toBe(false, 'the field should be
valid with ' + value);
+
+expect(isFieldInvalid(els.field)).toBe(true, 'the field should be
invalid with ' + value);
+expect(isSubmitDisabled(els.submit)).toBe(true, 'the submit button
should be disabled with ' + value);
+tearDown(els.field);
+ });
+});
+
+it('should keep the form enabled if the ip source field is valid', ()
=> {
+ const invalidValues = [
--- End diff --
Looks like this variable is misnamed, can you rename this (and it's usage
in at 382)to `validValues`?
> PCAP UI - Input validation
> --
>
> Key: METRON-1712
> URL: https://issues.apache.org/jira/browse/METRON-1712
> Project: Metron
> Issue Type: Sub-task
>Reporter: Shane Ardell
>Priority: Major
>
> The following fields need input validation:
> IP Source Address: should accept valid IPv4 address formats (v6 is not yet
> supported)
> IP Source Port: should accept integers from 0 to 65535
> IP Source Address: should accept valid IPv4 address formats (v6 is not yet
> supported)
> IP Dest Port: should accept integers from 0 to 65535
> Leave unvalidated:
> Protocol: number of valid protocols is around 150 and it could change in the
> future, validation possible makes more sense on the backend or by a separated
> endpoint
> Free text filtering: binary regular expression
> Broken inputs should be recognized by the UI and not submitted, with
> user-facing feedback. The REST API should also fail on queries outside
> reasonable parameters.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1143: METRON-1713: PCAP UI - Add a way to kill a pcap job
Github user tiborm commented on the issue: https://github.com/apache/metron/pull/1143 Thanks, Justin! I updated the PR with a small fix. ---
[jira] [Commented] (METRON-1713) PCAP UI - Add a way to kill a pcap job
[ https://issues.apache.org/jira/browse/METRON-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573231#comment-16573231 ] ASF GitHub Bot commented on METRON-1713: Github user tiborm commented on the issue: https://github.com/apache/metron/pull/1143 Thanks, Justin! I updated the PR with a small fix. > PCAP UI - Add a way to kill a pcap job > -- > > Key: METRON-1713 > URL: https://issues.apache.org/jira/browse/METRON-1713 > Project: Metron > Issue Type: Sub-task >Reporter: Shane Ardell >Priority: Major > Attachments: Screen Shot 2018-08-01 at 4.04.58 PM.png > > > As a user, I would like to be able to cancel my ongoing PCAP query request. > In order to do that, I would like to see a cancel button beside the progress > bar of the query. > An endpoint for this already exists in REST. > Attached image is from Swagger after building full-dev from the feature > branch from [METRON-1554|https://issues.apache.org/jira/browse/METRON-1554]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1132: METRON-1695: Expose pcap properties through Ambari
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1132 If you want to do the work in a separate PR (METRON-1709) then that's fine. As long as they are tested and committed together that works for me. If it were me, I would just do the work in this PR and save the trouble of managing two different PRs. We have several different components already that you can use as a template. I don't think this is that much work. Since this was developed against master I wouldn't switch to the feature branch. ---
[jira] [Commented] (METRON-1695) Expose pcap properties through Ambari
[ https://issues.apache.org/jira/browse/METRON-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573242#comment-16573242 ] ASF GitHub Bot commented on METRON-1695: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1132 If you want to do the work in a separate PR (METRON-1709) then that's fine. As long as they are tested and committed together that works for me. If it were me, I would just do the work in this PR and save the trouble of managing two different PRs. We have several different components already that you can use as a template. I don't think this is that much work. Since this was developed against master I wouldn't switch to the feature branch. > Expose pcap properties through Ambari > - > > Key: METRON-1695 > URL: https://issues.apache.org/jira/browse/METRON-1695 > Project: Metron > Issue Type: Bug >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > Currently, the $METRON_HOME/config/pcap.properties file is hardcoded with the > defaults. One has to hand edit the file before deploying the PCAP topology. > These properties should be configurable via Ambari. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1725) Add ability to specify YARN queue for pcap jobs
[ https://issues.apache.org/jira/browse/METRON-1725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman reassigned METRON-1725: - Assignee: Ryan Merriman > Add ability to specify YARN queue for pcap jobs > --- > > Key: METRON-1725 > URL: https://issues.apache.org/jira/browse/METRON-1725 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > There should be a way to configure which YARN queue a pcap query job is > submitted to. This option should be available for both the CLI and REST. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1722) PcapCLI should print progress to stdout
[ https://issues.apache.org/jira/browse/METRON-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman reassigned METRON-1722: - Assignee: Ryan Merriman > PcapCLI should print progress to stdout > --- > > Key: METRON-1722 > URL: https://issues.apache.org/jira/browse/METRON-1722 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > It would be helpful if the Pcap CLI reported job status included state and > percent complete. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1143: METRON-1713: PCAP UI - Add a way to kill a pcap job
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/1143
Hey, I played around with this a bit. I think there's more nuance here.
When I do a kill a job in the middle of submission, what happens is it kicks
off a kill that fails still:
```
{"timestamp":"2018-08-07 22:12:29","status":405,"error":"Method Not
Allowed","message":"Request method 'DELETE' not
supported","path":"/api/v1/pcap/kill/"}
```
The job still kicks off (which doesn't terribly surprise me, since it's
async). At that point, the progress bar has disappeared, but you can't kick
off another job (it'll throw an error since you're already running a job).
It seems like something like this (and feel free to throw out more ideas)
would be ncie:
1. The UI indicates cancellation of submitted job is in progress. E.g. the
progress bar or something similar to how error messages are displayed says
"Cancellation in progress". I'm not really sure what the best way to to it is.
2. The call to create a job waits until it gets back it's job id. It then
calls kill on the returned job. That way we aren't running the job in the
background and locking the user out.
3. After cancellation of the submitting job, the user is again free to
submit a new job. They're only locked out for the duration of the
cancellation, not until job completion.
---
[jira] [Commented] (METRON-1713) PCAP UI - Add a way to kill a pcap job
[
https://issues.apache.org/jira/browse/METRON-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573265#comment-16573265
]
ASF GitHub Bot commented on METRON-1713:
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/1143
Hey, I played around with this a bit. I think there's more nuance here.
When I do a kill a job in the middle of submission, what happens is it kicks
off a kill that fails still:
```
{"timestamp":"2018-08-07 22:12:29","status":405,"error":"Method Not
Allowed","message":"Request method 'DELETE' not
supported","path":"/api/v1/pcap/kill/"}
```
The job still kicks off (which doesn't terribly surprise me, since it's
async). At that point, the progress bar has disappeared, but you can't kick
off another job (it'll throw an error since you're already running a job).
It seems like something like this (and feel free to throw out more ideas)
would be ncie:
1. The UI indicates cancellation of submitted job is in progress. E.g. the
progress bar or something similar to how error messages are displayed says
"Cancellation in progress". I'm not really sure what the best way to to it is.
2. The call to create a job waits until it gets back it's job id. It then
calls kill on the returned job. That way we aren't running the job in the
background and locking the user out.
3. After cancellation of the submitting job, the user is again free to
submit a new job. They're only locked out for the duration of the
cancellation, not until job completion.
> PCAP UI - Add a way to kill a pcap job
> --
>
> Key: METRON-1713
> URL: https://issues.apache.org/jira/browse/METRON-1713
> Project: Metron
> Issue Type: Sub-task
>Reporter: Shane Ardell
>Priority: Major
> Attachments: Screen Shot 2018-08-01 at 4.04.58 PM.png
>
>
> As a user, I would like to be able to cancel my ongoing PCAP query request.
> In order to do that, I would like to see a cancel button beside the progress
> bar of the query.
> An endpoint for this already exists in REST.
> Attached image is from Swagger after building full-dev from the feature
> branch from [METRON-1554|https://issues.apache.org/jira/browse/METRON-1554].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1730) Update steps to run pycapa on Centos 6
[
https://issues.apache.org/jira/browse/METRON-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573494#comment-16573494
]
ASF GitHub Bot commented on METRON-1730:
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/1152#discussion_r208657384
--- Diff: metron-sensors/pycapa/README.md ---
@@ -76,8 +79,83 @@ General notes on the installation of Pycapa.
python setup.py install
```
+### Centos 6
+
+* These instructions can be used directly on CentOS 6 - useful for
developers using the Full Dev Vagrant test box.
+* Older distributions, like CentOS 6, that come with Python 2.6 installed,
should install Python 2.7 within a virtual environment and then run Pycapa from
within the virtual environment.
+
+1. Set up a couple environment variables.
+
+```
+PYCAPA_HOME=/opt/pycapa
+PYTHON27_HOME=/opt/rh/python27/root
+```
+
+1. Install required packages.
+
+```
+for item in epel-release centos-release-scl "@Development tools"
python27 python27-scldevel python27-python-virtualenv libpcap-devel
libselinux-python; do yum install -y $item; done
+```
+
+1. Setup Pycapa directory.
+
+```
+mkdir $PYCAPA_HOME && chmod 755 $PYCAPA_HOME
+```
+
+1. Create the virtualenv.
+
+```
+export LD_LIBRARY_PATH="/opt/rh/python27/root/usr/lib64"
+cd $PYCAPA_HOME
+${PYTHON27_HOME}/usr/bin/virtualenv pycapa-venv
+```
+
+1. Install Librdkafka at your chosen $PREFIX.
+
+```
+export PREFIX=/usr
+wget https://github.com/edenhill/librdkafka/archive/v0.11.5.tar.gz
-O - | tar -xz
--- End diff --
A problem I had recently setting this up was that librdkafka had stayed the
same, but the deps in dependencies.txt had been updated. Should we update the
requirements.txt to avoid this getting out of sync again?
> Update steps to run pycapa on Centos 6
> --
>
> Key: METRON-1730
> URL: https://issues.apache.org/jira/browse/METRON-1730
> Project: Metron
> Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1152: METRON-1730: Update steps to run pycapa on Centos...
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/1152#discussion_r208657384
--- Diff: metron-sensors/pycapa/README.md ---
@@ -76,8 +79,83 @@ General notes on the installation of Pycapa.
python setup.py install
```
+### Centos 6
+
+* These instructions can be used directly on CentOS 6 - useful for
developers using the Full Dev Vagrant test box.
+* Older distributions, like CentOS 6, that come with Python 2.6 installed,
should install Python 2.7 within a virtual environment and then run Pycapa from
within the virtual environment.
+
+1. Set up a couple environment variables.
+
+```
+PYCAPA_HOME=/opt/pycapa
+PYTHON27_HOME=/opt/rh/python27/root
+```
+
+1. Install required packages.
+
+```
+for item in epel-release centos-release-scl "@Development tools"
python27 python27-scldevel python27-python-virtualenv libpcap-devel
libselinux-python; do yum install -y $item; done
+```
+
+1. Setup Pycapa directory.
+
+```
+mkdir $PYCAPA_HOME && chmod 755 $PYCAPA_HOME
+```
+
+1. Create the virtualenv.
+
+```
+export LD_LIBRARY_PATH="/opt/rh/python27/root/usr/lib64"
+cd $PYCAPA_HOME
+${PYTHON27_HOME}/usr/bin/virtualenv pycapa-venv
+```
+
+1. Install Librdkafka at your chosen $PREFIX.
+
+```
+export PREFIX=/usr
+wget https://github.com/edenhill/librdkafka/archive/v0.11.5.tar.gz
-O - | tar -xz
--- End diff --
A problem I had recently setting this up was that librdkafka had stayed the
same, but the deps in dependencies.txt had been updated. Should we update the
requirements.txt to avoid this getting out of sync again?
---
[GitHub] metron pull request #1154: METRON-1720: Better error messages when there are...
GitHub user merrimanr opened a pull request: https://github.com/apache/metron/pull/1154 METRON-1720: Better error messages when there are no results or wireshark is not installed ## Contributor Comments This PR adds error handling when pdml results are requested after a pcap job has finished. I also added a small fix to a pcap-panel-component.ts test where the job status wasn't properly set. ### Testing This has been tested in full dev: 1. Navigate to the Pcap tab in the Alerts UI 2. Submit a Pcap query with a filter you know won't match any results 3. After the job finishes you should see an error message that says "No results found" 4. Uninstall wireshark in full dev: `yum -y remove wireshark` 5. Run a query that will produce results 6. You should get an error message that includes a message about tshark not being found ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/merrimanr/incubator-metron METRON-1720 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1154.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1154 commit 872d1b1ee13e358c18956945d71d3667d19fca8a Author: merrimanr Date: 2018-04-12T14:57:48Z Merge branch 'pcap-front' of https://github.com/simonellistonball/metron into pcaprest Conflicts: metron-interface/metron-alerts/src/app/app.module.ts commit b1b6a7dabea1a1d0d132482c8d97af29c0ac2683 Author: merrimanr Date: 2018-04-13T15:00:15Z initial commit Conflicts: metron-interface/metron-rest/src/main/java/org/apache/metron/rest/MetronRestApplication.java metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/PcapQueryController.java metron-interface/metron-rest/src/main/java/org/apache/metron/rest/util/pcapQueryThread.java commit 55cf2d945a4fcff1e7e2e47a234037ed6f394b2e Author: merrimanr Date: 2018-04-18T15:52:56Z added license headers commit 70696d047c6ef4b8ce5fcda03588474ff5b2c506 Author: tiborm Date
[jira] [Commented] (METRON-1720) Better error messages when there are no results or wireshark is not installed
[ https://issues.apache.org/jira/browse/METRON-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573546#comment-16573546 ] ASF GitHub Bot commented on METRON-1720: GitHub user merrimanr opened a pull request: https://github.com/apache/metron/pull/1154 METRON-1720: Better error messages when there are no results or wireshark is not installed ## Contributor Comments This PR adds error handling when pdml results are requested after a pcap job has finished. I also added a small fix to a pcap-panel-component.ts test where the job status wasn't properly set. ### Testing This has been tested in full dev: 1. Navigate to the Pcap tab in the Alerts UI 2. Submit a Pcap query with a filter you know won't match any results 3. After the job finishes you should see an error message that says "No results found" 4. Uninstall wireshark in full dev: `yum -y remove wireshark` 5. Run a query that will produce results 6. You should get an error message that includes a message about tshark not being found ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/merrimanr/incubator-metron METRON-1720 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1154.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1154 commit 872d1b1ee13e358c18956945d71d3667d19fca8a Author: merrimanr Date: 2018-04-12T14:57:48Z Merge branch 'pcap-front' of https://github.com/simonellistonball/metron into pcaprest Conflicts: metron-interface/metron-alerts/src/app/app.module.ts commit b1b6a7dabea1a1d0d132482c8d97af29c0ac2683 Author: merrimanr Date: 2018-04-13T15:00:15Z initial commit Conflicts: metron-interface/metron-rest/src/main/java/org/apache/metron/rest/MetronRestApplication.java metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/PcapQueryController.java metron-interface/metron-r
[jira] [Assigned] (METRON-1720) Better error messages when there are no results or wireshark is not installed
[ https://issues.apache.org/jira/browse/METRON-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman reassigned METRON-1720: - Assignee: Ryan Merriman > Better error messages when there are no results or wireshark is not installed > - > > Key: METRON-1720 > URL: https://issues.apache.org/jira/browse/METRON-1720 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > We should report why pcap data cannot be retrieved in pdml format after a > query has run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1154: METRON-1720: Better error messages when there are...
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1154#discussion_r208673702
--- Diff:
metron-interface/metron-alerts/src/app/pcap/service/pcap.service.ts ---
@@ -56,8 +56,7 @@ export class PcapService {
public getPackets(id: string, pageId: number): Observable {
return this.http.get(`/api/v1/pcap/${id}/pdml?page=${pageId}`, new
RequestOptions({headers: new Headers(this.defaultHeaders)}))
.map(HttpUtil.extractData)
-.catch(HttpUtil.handleError)
-.onErrorResumeNext();
--- End diff --
What are the implications of removing `onErrorResumeNext()`?
---
[jira] [Commented] (METRON-1720) Better error messages when there are no results or wireshark is not installed
[
https://issues.apache.org/jira/browse/METRON-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573579#comment-16573579
]
ASF GitHub Bot commented on METRON-1720:
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1154#discussion_r208673702
--- Diff:
metron-interface/metron-alerts/src/app/pcap/service/pcap.service.ts ---
@@ -56,8 +56,7 @@ export class PcapService {
public getPackets(id: string, pageId: number): Observable {
return this.http.get(`/api/v1/pcap/${id}/pdml?page=${pageId}`, new
RequestOptions({headers: new Headers(this.defaultHeaders)}))
.map(HttpUtil.extractData)
-.catch(HttpUtil.handleError)
-.onErrorResumeNext();
--- End diff --
What are the implications of removing `onErrorResumeNext()`?
> Better error messages when there are no results or wireshark is not installed
> -
>
> Key: METRON-1720
> URL: https://issues.apache.org/jira/browse/METRON-1720
> Project: Metron
> Issue Type: Sub-task
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
>
> We should report why pcap data cannot be retrieved in pdml format after a
> query has run.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1154: METRON-1720: Better error messages when there are...
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/1154#discussion_r208675839
--- Diff:
metron-interface/metron-alerts/src/app/pcap/service/pcap.service.ts ---
@@ -56,8 +56,7 @@ export class PcapService {
public getPackets(id: string, pageId: number): Observable {
return this.http.get(`/api/v1/pcap/${id}/pdml?page=${pageId}`, new
RequestOptions({headers: new Headers(this.defaultHeaders)}))
.map(HttpUtil.extractData)
-.catch(HttpUtil.handleError)
-.onErrorResumeNext();
--- End diff --
It won't catch the error if that line is included.
---
[jira] [Commented] (METRON-1720) Better error messages when there are no results or wireshark is not installed
[
https://issues.apache.org/jira/browse/METRON-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573587#comment-16573587
]
ASF GitHub Bot commented on METRON-1720:
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/1154#discussion_r208675839
--- Diff:
metron-interface/metron-alerts/src/app/pcap/service/pcap.service.ts ---
@@ -56,8 +56,7 @@ export class PcapService {
public getPackets(id: string, pageId: number): Observable {
return this.http.get(`/api/v1/pcap/${id}/pdml?page=${pageId}`, new
RequestOptions({headers: new Headers(this.defaultHeaders)}))
.map(HttpUtil.extractData)
-.catch(HttpUtil.handleError)
-.onErrorResumeNext();
--- End diff --
It won't catch the error if that line is included.
> Better error messages when there are no results or wireshark is not installed
> -
>
> Key: METRON-1720
> URL: https://issues.apache.org/jira/browse/METRON-1720
> Project: Metron
> Issue Type: Sub-task
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
>
> We should report why pcap data cannot be retrieved in pdml format after a
> query has run.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1154: METRON-1720: Better error messages when there are...
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1154#discussion_r208677034
--- Diff:
metron-interface/metron-alerts/src/app/pcap/service/pcap.service.ts ---
@@ -56,8 +56,7 @@ export class PcapService {
public getPackets(id: string, pageId: number): Observable {
return this.http.get(`/api/v1/pcap/${id}/pdml?page=${pageId}`, new
RequestOptions({headers: new Headers(this.defaultHeaders)}))
.map(HttpUtil.extractData)
-.catch(HttpUtil.handleError)
-.onErrorResumeNext();
--- End diff --
So previously it would catch the error and continue on regardless. Now it
catches the error and handles it?
---
[jira] [Commented] (METRON-1720) Better error messages when there are no results or wireshark is not installed
[
https://issues.apache.org/jira/browse/METRON-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573592#comment-16573592
]
ASF GitHub Bot commented on METRON-1720:
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1154#discussion_r208677034
--- Diff:
metron-interface/metron-alerts/src/app/pcap/service/pcap.service.ts ---
@@ -56,8 +56,7 @@ export class PcapService {
public getPackets(id: string, pageId: number): Observable {
return this.http.get(`/api/v1/pcap/${id}/pdml?page=${pageId}`, new
RequestOptions({headers: new Headers(this.defaultHeaders)}))
.map(HttpUtil.extractData)
-.catch(HttpUtil.handleError)
-.onErrorResumeNext();
--- End diff --
So previously it would catch the error and continue on regardless. Now it
catches the error and handles it?
> Better error messages when there are no results or wireshark is not installed
> -
>
> Key: METRON-1720
> URL: https://issues.apache.org/jira/browse/METRON-1720
> Project: Metron
> Issue Type: Sub-task
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
>
> We should report why pcap data cannot be retrieved in pdml format after a
> query has run.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1151: METRON-1728: Handle null values in config in Pcap...
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/1151#discussion_r208679310
--- Diff:
metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/mr/PcapJob.java
---
@@ -216,20 +218,22 @@ public void setCompleteCheckInterval(long interval) {
Configuration hadoopConf = PcapOptions.HADOOP_CONF.get(configuration,
Configuration.class);
FileSystem fileSystem = PcapOptions.FILESYSTEM.get(configuration,
FileSystem.class);
Path basePath = PcapOptions.BASE_PATH.getTransformed(configuration,
Path.class);
-Path baseInterimResultPath =
PcapOptions.BASE_INTERIM_RESULT_PATH.getTransformed(configuration, Path.class);
+Path baseInterimResultPath = PcapOptions.BASE_INTERIM_RESULT_PATH
+.getTransformedOrDefault(configuration, Path.class,
+new Path(PcapGlobalDefaults.BASE_INTERIM_RESULT_PATH_DEFAULT));
long startTime;
if (configuration.containsKey(PcapOptions.START_TIME_NS.getKey())) {
- startTime = PcapOptions.START_TIME_NS.get(configuration, Long.class);
+ startTime = PcapOptions.START_TIME_NS.getOrDefault(configuration,
Long.class, 0L);
} else {
- startTime = PcapOptions.START_TIME_MS.get(configuration, Long.class)
* 100;
+ startTime = PcapOptions.START_TIME_MS.getOrDefault(configuration,
Long.class, 0L) * 100;
}
long endTime;
if (configuration.containsKey(PcapOptions.END_TIME_NS.getKey())) {
- endTime = PcapOptions.END_TIME_NS.get(configuration, Long.class);
+ endTime = PcapOptions.END_TIME_NS.getOrDefault(configuration,
Long.class, System.nanoTime());
--- End diff --
Nanotime isn't actually current time in nanos or anything objectively
meaningful (and can vary by jvm or be negative or whatever). Check out
https://docs.oracle.com/javase/7/docs/api/java/lang/System.html#nanoTime()
Particularly,
> This method can only be used to measure elapsed time and is not related
to any other notion of system or wall-clock time. The value returned represents
nanoseconds since some fixed but arbitrary origin time (perhaps in the future,
so values may be negative).
I'd just make it System.currentTimeMillis() and adjust, since I don't think
there's a clean way to get a real nanos timestamp in Java (at least until 9,
maybe)
---
[jira] [Commented] (METRON-1728) Handle null values in config in Pcap backend more gracefully
[
https://issues.apache.org/jira/browse/METRON-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573607#comment-16573607
]
ASF GitHub Bot commented on METRON-1728:
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/1151#discussion_r208679310
--- Diff:
metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/mr/PcapJob.java
---
@@ -216,20 +218,22 @@ public void setCompleteCheckInterval(long interval) {
Configuration hadoopConf = PcapOptions.HADOOP_CONF.get(configuration,
Configuration.class);
FileSystem fileSystem = PcapOptions.FILESYSTEM.get(configuration,
FileSystem.class);
Path basePath = PcapOptions.BASE_PATH.getTransformed(configuration,
Path.class);
-Path baseInterimResultPath =
PcapOptions.BASE_INTERIM_RESULT_PATH.getTransformed(configuration, Path.class);
+Path baseInterimResultPath = PcapOptions.BASE_INTERIM_RESULT_PATH
+.getTransformedOrDefault(configuration, Path.class,
+new Path(PcapGlobalDefaults.BASE_INTERIM_RESULT_PATH_DEFAULT));
long startTime;
if (configuration.containsKey(PcapOptions.START_TIME_NS.getKey())) {
- startTime = PcapOptions.START_TIME_NS.get(configuration, Long.class);
+ startTime = PcapOptions.START_TIME_NS.getOrDefault(configuration,
Long.class, 0L);
} else {
- startTime = PcapOptions.START_TIME_MS.get(configuration, Long.class)
* 100;
+ startTime = PcapOptions.START_TIME_MS.getOrDefault(configuration,
Long.class, 0L) * 100;
}
long endTime;
if (configuration.containsKey(PcapOptions.END_TIME_NS.getKey())) {
- endTime = PcapOptions.END_TIME_NS.get(configuration, Long.class);
+ endTime = PcapOptions.END_TIME_NS.getOrDefault(configuration,
Long.class, System.nanoTime());
--- End diff --
Nanotime isn't actually current time in nanos or anything objectively
meaningful (and can vary by jvm or be negative or whatever). Check out
https://docs.oracle.com/javase/7/docs/api/java/lang/System.html#nanoTime()
Particularly,
> This method can only be used to measure elapsed time and is not related
to any other notion of system or wall-clock time. The value returned represents
nanoseconds since some fixed but arbitrary origin time (perhaps in the future,
so values may be negative).
I'd just make it System.currentTimeMillis() and adjust, since I don't think
there's a clean way to get a real nanos timestamp in Java (at least until 9,
maybe)
> Handle null values in config in Pcap backend more gracefully
>
>
> Key: METRON-1728
> URL: https://issues.apache.org/jira/browse/METRON-1728
> Project: Metron
> Issue Type: Sub-task
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1152: METRON-1730: Update steps to run pycapa on Centos...
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1152#discussion_r208683126
--- Diff: metron-sensors/pycapa/README.md ---
@@ -76,8 +79,83 @@ General notes on the installation of Pycapa.
python setup.py install
```
+### Centos 6
+
+* These instructions can be used directly on CentOS 6 - useful for
developers using the Full Dev Vagrant test box.
+* Older distributions, like CentOS 6, that come with Python 2.6 installed,
should install Python 2.7 within a virtual environment and then run Pycapa from
within the virtual environment.
+
+1. Set up a couple environment variables.
+
+```
+PYCAPA_HOME=/opt/pycapa
+PYTHON27_HOME=/opt/rh/python27/root
+```
+
+1. Install required packages.
+
+```
+for item in epel-release centos-release-scl "@Development tools"
python27 python27-scldevel python27-python-virtualenv libpcap-devel
libselinux-python; do yum install -y $item; done
+```
+
+1. Setup Pycapa directory.
+
+```
+mkdir $PYCAPA_HOME && chmod 755 $PYCAPA_HOME
+```
+
+1. Create the virtualenv.
+
+```
+export LD_LIBRARY_PATH="/opt/rh/python27/root/usr/lib64"
+cd $PYCAPA_HOME
+${PYTHON27_HOME}/usr/bin/virtualenv pycapa-venv
+```
+
+1. Install Librdkafka at your chosen $PREFIX.
+
+```
+export PREFIX=/usr
+wget https://github.com/edenhill/librdkafka/archive/v0.11.5.tar.gz
-O - | tar -xz
--- End diff --
@justinleet was that as simple as using the current version of the deps and
declaring it as `mydep==x.y.z` in requirements.txt?
---
[jira] [Commented] (METRON-1730) Update steps to run pycapa on Centos 6
[
https://issues.apache.org/jira/browse/METRON-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573635#comment-16573635
]
ASF GitHub Bot commented on METRON-1730:
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1152#discussion_r208683126
--- Diff: metron-sensors/pycapa/README.md ---
@@ -76,8 +79,83 @@ General notes on the installation of Pycapa.
python setup.py install
```
+### Centos 6
+
+* These instructions can be used directly on CentOS 6 - useful for
developers using the Full Dev Vagrant test box.
+* Older distributions, like CentOS 6, that come with Python 2.6 installed,
should install Python 2.7 within a virtual environment and then run Pycapa from
within the virtual environment.
+
+1. Set up a couple environment variables.
+
+```
+PYCAPA_HOME=/opt/pycapa
+PYTHON27_HOME=/opt/rh/python27/root
+```
+
+1. Install required packages.
+
+```
+for item in epel-release centos-release-scl "@Development tools"
python27 python27-scldevel python27-python-virtualenv libpcap-devel
libselinux-python; do yum install -y $item; done
+```
+
+1. Setup Pycapa directory.
+
+```
+mkdir $PYCAPA_HOME && chmod 755 $PYCAPA_HOME
+```
+
+1. Create the virtualenv.
+
+```
+export LD_LIBRARY_PATH="/opt/rh/python27/root/usr/lib64"
+cd $PYCAPA_HOME
+${PYTHON27_HOME}/usr/bin/virtualenv pycapa-venv
+```
+
+1. Install Librdkafka at your chosen $PREFIX.
+
+```
+export PREFIX=/usr
+wget https://github.com/edenhill/librdkafka/archive/v0.11.5.tar.gz
-O - | tar -xz
--- End diff --
@justinleet was that as simple as using the current version of the deps and
declaring it as `mydep==x.y.z` in requirements.txt?
> Update steps to run pycapa on Centos 6
> --
>
> Key: METRON-1730
> URL: https://issues.apache.org/jira/browse/METRON-1730
> Project: Metron
> Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1154: METRON-1720: Better error messages when there are...
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/1154#discussion_r208690899
--- Diff:
metron-interface/metron-alerts/src/app/pcap/service/pcap.service.ts ---
@@ -56,8 +56,7 @@ export class PcapService {
public getPackets(id: string, pageId: number): Observable {
return this.http.get(`/api/v1/pcap/${id}/pdml?page=${pageId}`, new
RequestOptions({headers: new Headers(this.defaultHeaders)}))
.map(HttpUtil.extractData)
-.catch(HttpUtil.handleError)
-.onErrorResumeNext();
--- End diff --
It was not passing the error on to the observable.
---
[jira] [Commented] (METRON-1720) Better error messages when there are no results or wireshark is not installed
[
https://issues.apache.org/jira/browse/METRON-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573683#comment-16573683
]
ASF GitHub Bot commented on METRON-1720:
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/1154#discussion_r208690899
--- Diff:
metron-interface/metron-alerts/src/app/pcap/service/pcap.service.ts ---
@@ -56,8 +56,7 @@ export class PcapService {
public getPackets(id: string, pageId: number): Observable {
return this.http.get(`/api/v1/pcap/${id}/pdml?page=${pageId}`, new
RequestOptions({headers: new Headers(this.defaultHeaders)}))
.map(HttpUtil.extractData)
-.catch(HttpUtil.handleError)
-.onErrorResumeNext();
--- End diff --
It was not passing the error on to the observable.
> Better error messages when there are no results or wireshark is not installed
> -
>
> Key: METRON-1720
> URL: https://issues.apache.org/jira/browse/METRON-1720
> Project: Metron
> Issue Type: Sub-task
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
>
> We should report why pcap data cannot be retrieved in pdml format after a
> query has run.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1134: METRON-1696: Create the HDFS directory for pcap sequence...
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1134 @anandsubbu @MohanDV - if we set this to metron:hadoop will that cover both cases? If it does, I would request you make this change in the feature branch. It should work for both kerberized/non-kerberized setups. ---
[jira] [Commented] (METRON-1696) Pcap parser fails to write pacap sequence file to hdfs on kerberized cluster
[
https://issues.apache.org/jira/browse/METRON-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573686#comment-16573686
]
ASF GitHub Bot commented on METRON-1696:
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/1134
@anandsubbu @MohanDV - if we set this to metron:hadoop will that cover both
cases? If it does, I would request you make this change in the feature branch.
It should work for both kerberized/non-kerberized setups.
> Pcap parser fails to write pacap sequence file to hdfs on kerberized cluster
> -
>
> Key: METRON-1696
> URL: https://issues.apache.org/jira/browse/METRON-1696
> Project: Metron
> Issue Type: Bug
>Reporter: Mohan
>Assignee: Mohan
>Priority: Major
>
> pcap parser fails to write the pcap sequence files to hdfs directory due to
> insufficient privileges to hdfs folder for 'metron' user
> {code:java}
> 2018-07-25 10:15:50.035 o.a.m.s.p.HDFSWriterCallback
> Thread-9-kafkaSpout-executor[3 3] [ERROR] Permission denied: user=metron,
> access=WRITE,
> inode="/apps/metron/pcap/pcap_pcap_1532513746365022000_0_pcap-20-1532414055":hdfs:hdfs:drwxr-xr-x
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:353)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:325)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:246)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1950)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1934)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1917)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2767)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2702)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2586)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:736)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:409)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1720) Better error messages when there are no results or wireshark is not installed
[ https://issues.apache.org/jira/browse/METRON-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573692#comment-16573692 ] ASF GitHub Bot commented on METRON-1720: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1154 @merrimanr thanks for the clarification. +1 via inspection. > Better error messages when there are no results or wireshark is not installed > - > > Key: METRON-1720 > URL: https://issues.apache.org/jira/browse/METRON-1720 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > We should report why pcap data cannot be retrieved in pdml format after a > query has run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1154: METRON-1720: Better error messages when there are no res...
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1154 @merrimanr thanks for the clarification. +1 via inspection. ---
[GitHub] metron issue #1139: METRON-1723: PCAP UI - Unable to select/copy from packet...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1139 Works as advertised. +1 ---
[jira] [Commented] (METRON-1723) PCAP UI - Unable to select/copy from packets details in PCAP query panel
[ https://issues.apache.org/jira/browse/METRON-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573694#comment-16573694 ] ASF GitHub Bot commented on METRON-1723: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1139 Works as advertised. +1 > PCAP UI - Unable to select/copy from packets details in PCAP query panel > > > Key: METRON-1723 > URL: https://issues.apache.org/jira/browse/METRON-1723 > Project: Metron > Issue Type: Sub-task >Reporter: Shane Ardell >Priority: Major > Attachments: expand-collapse-details-PCAP-query.mp4 > > > Upon query completion, a user can click on a row and drill down (up to 2 > levels) for a given packet. > A SOC analyst might find it useful to select/copy specific information from > the 2nd level granular details (E.g. MAC address, IP address, timestamps > etc.) to be used subsequently for analysis/investigation. > Currently, selecting or clicking on nested content fires an event that > expands/collapses the information. See the attached video to view the > behavior. > It would be better if target for the collapse/expand event only applied to > the top of the details section. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1712) PCAP UI - Input validation
[ https://issues.apache.org/jira/browse/METRON-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573703#comment-16573703 ] ASF GitHub Bot commented on METRON-1712: Github user tiborm commented on the issue: https://github.com/apache/metron/pull/1142 You're right. Copy paste issue. Fixed. Thanks! > PCAP UI - Input validation > -- > > Key: METRON-1712 > URL: https://issues.apache.org/jira/browse/METRON-1712 > Project: Metron > Issue Type: Sub-task >Reporter: Shane Ardell >Priority: Major > > The following fields need input validation: > IP Source Address: should accept valid IPv4 address formats (v6 is not yet > supported) > IP Source Port: should accept integers from 0 to 65535 > IP Source Address: should accept valid IPv4 address formats (v6 is not yet > supported) > IP Dest Port: should accept integers from 0 to 65535 > Leave unvalidated: > Protocol: number of valid protocols is around 150 and it could change in the > future, validation possible makes more sense on the backend or by a separated > endpoint > Free text filtering: binary regular expression > Broken inputs should be recognized by the UI and not submitted, with > user-facing feedback. The REST API should also fail on queries outside > reasonable parameters. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1142: METRON-1712: PCAP UI Input validation
Github user tiborm commented on the issue: https://github.com/apache/metron/pull/1142 You're right. Copy paste issue. Fixed. Thanks! ---
[GitHub] metron pull request #1142: METRON-1712: PCAP UI Input validation
Github user tiborm commented on a diff in the pull request:
https://github.com/apache/metron/pull/1142#discussion_r208695454
--- Diff:
metron-interface/metron-alerts/src/app/pcap/pcap-filters/pcap-filters.component.spec.ts
---
@@ -331,13 +337,60 @@ describe('PcapFiltersComponent', () => {
validValues.forEach((value) => {
const els = getFieldWithSubmit('ip-dest-port');
+expect(isFieldInvalid(els.field)).toBe(false, 'the field should be
valid without ' + value);
+expect(isSubmitDisabled(els.submit)).toBe(false, 'the submit
button should be enabled without ' + value);
+
+setFieldValue(els.field, value);
+
expect(isFieldInvalid(els.field)).toBe(false, 'the field should be
valid with ' + value);
expect(isSubmitDisabled(els.submit)).toBe(false, 'the submit
button should be enabled with ' + value);
+tearDown(els.field);
+ });
+});
+
+
+it('should disable the form if the ip source field is invalid', () => {
+ const invalidValues = [
+'tst',
+0o0,
+0,
+'111.111.111',
+'222.222.222.222.222',
+'333.333.333.333',
+ ];
+
+ invalidValues.forEach((value) => {
+const els = getFieldWithSubmit('ip-src-addr');
+expect(isFieldInvalid(els.field)).toBe(false, 'the field should be
valid without ' + value);
+expect(isSubmitDisabled(els.submit)).toBe(false, 'the submit
button should be enabled without ' + value);
+
setFieldValue(els.field, value);
-expect(isFieldInvalid(els.field)).toBe(false, 'the field should be
valid with ' + value);
+
+expect(isFieldInvalid(els.field)).toBe(true, 'the field should be
invalid with ' + value);
+expect(isSubmitDisabled(els.submit)).toBe(true, 'the submit button
should be disabled with ' + value);
+tearDown(els.field);
+ });
+});
+
+it('should keep the form enabled if the ip source field is valid', ()
=> {
+ const invalidValues = [
--- End diff --
Fixed. Thanks!
---
[jira] [Commented] (METRON-1712) PCAP UI - Input validation
[
https://issues.apache.org/jira/browse/METRON-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573704#comment-16573704
]
ASF GitHub Bot commented on METRON-1712:
Github user tiborm commented on a diff in the pull request:
https://github.com/apache/metron/pull/1142#discussion_r208695454
--- Diff:
metron-interface/metron-alerts/src/app/pcap/pcap-filters/pcap-filters.component.spec.ts
---
@@ -331,13 +337,60 @@ describe('PcapFiltersComponent', () => {
validValues.forEach((value) => {
const els = getFieldWithSubmit('ip-dest-port');
+expect(isFieldInvalid(els.field)).toBe(false, 'the field should be
valid without ' + value);
+expect(isSubmitDisabled(els.submit)).toBe(false, 'the submit
button should be enabled without ' + value);
+
+setFieldValue(els.field, value);
+
expect(isFieldInvalid(els.field)).toBe(false, 'the field should be
valid with ' + value);
expect(isSubmitDisabled(els.submit)).toBe(false, 'the submit
button should be enabled with ' + value);
+tearDown(els.field);
+ });
+});
+
+
+it('should disable the form if the ip source field is invalid', () => {
+ const invalidValues = [
+'tst',
+0o0,
+0,
+'111.111.111',
+'222.222.222.222.222',
+'333.333.333.333',
+ ];
+
+ invalidValues.forEach((value) => {
+const els = getFieldWithSubmit('ip-src-addr');
+expect(isFieldInvalid(els.field)).toBe(false, 'the field should be
valid without ' + value);
+expect(isSubmitDisabled(els.submit)).toBe(false, 'the submit
button should be enabled without ' + value);
+
setFieldValue(els.field, value);
-expect(isFieldInvalid(els.field)).toBe(false, 'the field should be
valid with ' + value);
+
+expect(isFieldInvalid(els.field)).toBe(true, 'the field should be
invalid with ' + value);
+expect(isSubmitDisabled(els.submit)).toBe(true, 'the submit button
should be disabled with ' + value);
+tearDown(els.field);
+ });
+});
+
+it('should keep the form enabled if the ip source field is valid', ()
=> {
+ const invalidValues = [
--- End diff --
Fixed. Thanks!
> PCAP UI - Input validation
> --
>
> Key: METRON-1712
> URL: https://issues.apache.org/jira/browse/METRON-1712
> Project: Metron
> Issue Type: Sub-task
>Reporter: Shane Ardell
>Priority: Major
>
> The following fields need input validation:
> IP Source Address: should accept valid IPv4 address formats (v6 is not yet
> supported)
> IP Source Port: should accept integers from 0 to 65535
> IP Source Address: should accept valid IPv4 address formats (v6 is not yet
> supported)
> IP Dest Port: should accept integers from 0 to 65535
> Leave unvalidated:
> Protocol: number of valid protocols is around 150 and it could change in the
> future, validation possible makes more sense on the backend or by a separated
> endpoint
> Free text filtering: binary regular expression
> Broken inputs should be recognized by the UI and not submitted, with
> user-facing feedback. The REST API should also fail on queries outside
> reasonable parameters.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1142: METRON-1712: PCAP UI Input validation
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1142 +1 pending Travis, thanks for the fixes, this is a great quality of life improvement! ---
[jira] [Commented] (METRON-1712) PCAP UI - Input validation
[ https://issues.apache.org/jira/browse/METRON-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573705#comment-16573705 ] ASF GitHub Bot commented on METRON-1712: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1142 +1 pending Travis, thanks for the fixes, this is a great quality of life improvement! > PCAP UI - Input validation > -- > > Key: METRON-1712 > URL: https://issues.apache.org/jira/browse/METRON-1712 > Project: Metron > Issue Type: Sub-task >Reporter: Shane Ardell >Priority: Major > > The following fields need input validation: > IP Source Address: should accept valid IPv4 address formats (v6 is not yet > supported) > IP Source Port: should accept integers from 0 to 65535 > IP Source Address: should accept valid IPv4 address formats (v6 is not yet > supported) > IP Dest Port: should accept integers from 0 to 65535 > Leave unvalidated: > Protocol: number of valid protocols is around 150 and it could change in the > future, validation possible makes more sense on the backend or by a separated > endpoint > Free text filtering: binary regular expression > Broken inputs should be recognized by the UI and not submitted, with > user-facing feedback. The REST API should also fail on queries outside > reasonable parameters. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1143: METRON-1713: PCAP UI - Add a way to kill a pcap job
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1143 Does this PR also contain METRON-1712? I don't see anything in the PR description stating that and I see several changes that are not related to this PR. ---
[jira] [Commented] (METRON-1713) PCAP UI - Add a way to kill a pcap job
[ https://issues.apache.org/jira/browse/METRON-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573709#comment-16573709 ] ASF GitHub Bot commented on METRON-1713: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1143 Does this PR also contain METRON-1712? I don't see anything in the PR description stating that and I see several changes that are not related to this PR. > PCAP UI - Add a way to kill a pcap job > -- > > Key: METRON-1713 > URL: https://issues.apache.org/jira/browse/METRON-1713 > Project: Metron > Issue Type: Sub-task >Reporter: Shane Ardell >Priority: Major > Attachments: Screen Shot 2018-08-01 at 4.04.58 PM.png > > > As a user, I would like to be able to cancel my ongoing PCAP query request. > In order to do that, I would like to see a cancel button beside the progress > bar of the query. > An endpoint for this already exists in REST. > Attached image is from Swagger after building full-dev from the feature > branch from [METRON-1554|https://issues.apache.org/jira/browse/METRON-1554]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1151: METRON-1728: Handle null values in config in Pcap...
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1151#discussion_r208697541
--- Diff:
metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/mr/PcapJob.java
---
@@ -216,20 +218,22 @@ public void setCompleteCheckInterval(long interval) {
Configuration hadoopConf = PcapOptions.HADOOP_CONF.get(configuration,
Configuration.class);
FileSystem fileSystem = PcapOptions.FILESYSTEM.get(configuration,
FileSystem.class);
Path basePath = PcapOptions.BASE_PATH.getTransformed(configuration,
Path.class);
-Path baseInterimResultPath =
PcapOptions.BASE_INTERIM_RESULT_PATH.getTransformed(configuration, Path.class);
+Path baseInterimResultPath = PcapOptions.BASE_INTERIM_RESULT_PATH
+.getTransformedOrDefault(configuration, Path.class,
+new Path(PcapGlobalDefaults.BASE_INTERIM_RESULT_PATH_DEFAULT));
long startTime;
if (configuration.containsKey(PcapOptions.START_TIME_NS.getKey())) {
- startTime = PcapOptions.START_TIME_NS.get(configuration, Long.class);
+ startTime = PcapOptions.START_TIME_NS.getOrDefault(configuration,
Long.class, 0L);
} else {
- startTime = PcapOptions.START_TIME_MS.get(configuration, Long.class)
* 100;
+ startTime = PcapOptions.START_TIME_MS.getOrDefault(configuration,
Long.class, 0L) * 100;
}
long endTime;
if (configuration.containsKey(PcapOptions.END_TIME_NS.getKey())) {
- endTime = PcapOptions.END_TIME_NS.get(configuration, Long.class);
+ endTime = PcapOptions.END_TIME_NS.getOrDefault(configuration,
Long.class, System.nanoTime());
--- End diff --
@justinleet good catch! Thanks.
---
[jira] [Commented] (METRON-1728) Handle null values in config in Pcap backend more gracefully
[
https://issues.apache.org/jira/browse/METRON-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573711#comment-16573711
]
ASF GitHub Bot commented on METRON-1728:
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1151#discussion_r208697541
--- Diff:
metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/mr/PcapJob.java
---
@@ -216,20 +218,22 @@ public void setCompleteCheckInterval(long interval) {
Configuration hadoopConf = PcapOptions.HADOOP_CONF.get(configuration,
Configuration.class);
FileSystem fileSystem = PcapOptions.FILESYSTEM.get(configuration,
FileSystem.class);
Path basePath = PcapOptions.BASE_PATH.getTransformed(configuration,
Path.class);
-Path baseInterimResultPath =
PcapOptions.BASE_INTERIM_RESULT_PATH.getTransformed(configuration, Path.class);
+Path baseInterimResultPath = PcapOptions.BASE_INTERIM_RESULT_PATH
+.getTransformedOrDefault(configuration, Path.class,
+new Path(PcapGlobalDefaults.BASE_INTERIM_RESULT_PATH_DEFAULT));
long startTime;
if (configuration.containsKey(PcapOptions.START_TIME_NS.getKey())) {
- startTime = PcapOptions.START_TIME_NS.get(configuration, Long.class);
+ startTime = PcapOptions.START_TIME_NS.getOrDefault(configuration,
Long.class, 0L);
} else {
- startTime = PcapOptions.START_TIME_MS.get(configuration, Long.class)
* 100;
+ startTime = PcapOptions.START_TIME_MS.getOrDefault(configuration,
Long.class, 0L) * 100;
}
long endTime;
if (configuration.containsKey(PcapOptions.END_TIME_NS.getKey())) {
- endTime = PcapOptions.END_TIME_NS.get(configuration, Long.class);
+ endTime = PcapOptions.END_TIME_NS.getOrDefault(configuration,
Long.class, System.nanoTime());
--- End diff --
@justinleet good catch! Thanks.
> Handle null values in config in Pcap backend more gracefully
>
>
> Key: METRON-1728
> URL: https://issues.apache.org/jira/browse/METRON-1728
> Project: Metron
> Issue Type: Sub-task
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (METRON-1671) Create PCAP UI
[ https://issues.apache.org/jira/browse/METRON-1671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic reassigned METRON-1671: - Assignee: Tibor Meller > Create PCAP UI > -- > > Key: METRON-1671 > URL: https://issues.apache.org/jira/browse/METRON-1671 > Project: Metron > Issue Type: Sub-task >Reporter: Tibor Meller >Assignee: Tibor Meller >Priority: Major > > As a user, I like to submit PCAP query requests from the Metron UI. > As a user, I like to be able to narrow down the result set of the query by > the following filters: > - IP Source Address > - IP Source Port > - IP Dest Address > - IP Dest Port > - Protocol > - Include Reverse Traffic > - Free text filtering > As a user, I like to get feedback from the progress of the ongoing request by > a progress bar. > As a user, I like to see the result of the query in a grid. Each line of the > grid should show and extendable packet record. > As a user, when I clicking on a row of the result grid I would like to see > additional information about that particular packet. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1713) PCAP UI - Add a way to kill a pcap job
[ https://issues.apache.org/jira/browse/METRON-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573724#comment-16573724 ] ASF GitHub Bot commented on METRON-1713: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1143 @justinleet I am seeing different behavior. When I submit a query, the cancel button appears but is disabled. I am not able to kill a job in the middle of submission. After the job submission request returns with a job id, the cancel button becomes active. When I click the cancel button, the kill command succeeds with no errors. If we want to add a "Cancellation in progress" message while the kill request is being processed I think that makes sense. For me, this is working as expected. The error you posted has nothing to do with the state of a job submission. My guess is that you don't have the most recent code deployed in full dev. That 405 error suggests that endpoint doesn't exist. Let me know if I am misunderstanding something. > PCAP UI - Add a way to kill a pcap job > -- > > Key: METRON-1713 > URL: https://issues.apache.org/jira/browse/METRON-1713 > Project: Metron > Issue Type: Sub-task >Reporter: Shane Ardell >Priority: Major > Attachments: Screen Shot 2018-08-01 at 4.04.58 PM.png > > > As a user, I would like to be able to cancel my ongoing PCAP query request. > In order to do that, I would like to see a cancel button beside the progress > bar of the query. > An endpoint for this already exists in REST. > Attached image is from Swagger after building full-dev from the feature > branch from [METRON-1554|https://issues.apache.org/jira/browse/METRON-1554]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1143: METRON-1713: PCAP UI - Add a way to kill a pcap job
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1143 @justinleet I am seeing different behavior. When I submit a query, the cancel button appears but is disabled. I am not able to kill a job in the middle of submission. After the job submission request returns with a job id, the cancel button becomes active. When I click the cancel button, the kill command succeeds with no errors. If we want to add a "Cancellation in progress" message while the kill request is being processed I think that makes sense. For me, this is working as expected. The error you posted has nothing to do with the state of a job submission. My guess is that you don't have the most recent code deployed in full dev. That 405 error suggests that endpoint doesn't exist. Let me know if I am misunderstanding something. ---
[GitHub] metron pull request #1154: METRON-1720: Better error messages when there are...
Github user merrimanr closed the pull request at: https://github.com/apache/metron/pull/1154 ---
[jira] [Commented] (METRON-1720) Better error messages when there are no results or wireshark is not installed
[ https://issues.apache.org/jira/browse/METRON-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573726#comment-16573726 ] ASF GitHub Bot commented on METRON-1720: Github user merrimanr closed the pull request at: https://github.com/apache/metron/pull/1154 > Better error messages when there are no results or wireshark is not installed > - > > Key: METRON-1720 > URL: https://issues.apache.org/jira/browse/METRON-1720 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > We should report why pcap data cannot be retrieved in pdml format after a > query has run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1143: METRON-1713: PCAP UI - Add a way to kill a pcap job
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1143 Sigh, my inability to get my browser in a consistent state strikes again. Clearing everything out and trying again shows the behavior you describe. ---
[jira] [Commented] (METRON-1713) PCAP UI - Add a way to kill a pcap job
[ https://issues.apache.org/jira/browse/METRON-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573732#comment-16573732 ] ASF GitHub Bot commented on METRON-1713: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1143 Sigh, my inability to get my browser in a consistent state strikes again. Clearing everything out and trying again shows the behavior you describe. > PCAP UI - Add a way to kill a pcap job > -- > > Key: METRON-1713 > URL: https://issues.apache.org/jira/browse/METRON-1713 > Project: Metron > Issue Type: Sub-task >Reporter: Shane Ardell >Priority: Major > Attachments: Screen Shot 2018-08-01 at 4.04.58 PM.png > > > As a user, I would like to be able to cancel my ongoing PCAP query request. > In order to do that, I would like to see a cancel button beside the progress > bar of the query. > An endpoint for this already exists in REST. > Attached image is from Swagger after building full-dev from the feature > branch from [METRON-1554|https://issues.apache.org/jira/browse/METRON-1554]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1143: METRON-1713: PCAP UI - Add a way to kill a pcap job
Github user tiborm commented on the issue: https://github.com/apache/metron/pull/1143 @merrimanr Yes, it contains the latest state of METRON-1712 and feature/METRON-1554-pcap-query-panel. ---
[jira] [Commented] (METRON-1713) PCAP UI - Add a way to kill a pcap job
[ https://issues.apache.org/jira/browse/METRON-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573833#comment-16573833 ] ASF GitHub Bot commented on METRON-1713: Github user tiborm commented on the issue: https://github.com/apache/metron/pull/1143 @merrimanr Yes, it contains the latest state of METRON-1712 and feature/METRON-1554-pcap-query-panel. > PCAP UI - Add a way to kill a pcap job > -- > > Key: METRON-1713 > URL: https://issues.apache.org/jira/browse/METRON-1713 > Project: Metron > Issue Type: Sub-task >Reporter: Shane Ardell >Priority: Major > Attachments: Screen Shot 2018-08-01 at 4.04.58 PM.png > > > As a user, I would like to be able to cancel my ongoing PCAP query request. > In order to do that, I would like to see a cancel button beside the progress > bar of the query. > An endpoint for this already exists in REST. > Attached image is from Swagger after building full-dev from the feature > branch from [METRON-1554|https://issues.apache.org/jira/browse/METRON-1554]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1731) PCAP - Escape colons in output dir names
Michael Miklavcic created METRON-1731: - Summary: PCAP - Escape colons in output dir names Key: METRON-1731 URL: https://issues.apache.org/jira/browse/METRON-1731 Project: Metron Issue Type: Improvement Reporter: Michael Miklavcic Assignee: Michael Miklavcic -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1731) PCAP - Escape colons in output dir names
[ https://issues.apache.org/jira/browse/METRON-1731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic updated METRON-1731: -- Issue Type: Sub-task (was: Improvement) Parent: METRON-1554 > PCAP - Escape colons in output dir names > > > Key: METRON-1731 > URL: https://issues.apache.org/jira/browse/METRON-1731 > Project: Metron > Issue Type: Sub-task >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1155: METRON-1731: PCAP - Escape colons in output dir n...
GitHub user mmiklavc opened a pull request: https://github.com/apache/metron/pull/1155 METRON-1731: PCAP - Escape colons in output dir names ## Contributor Comments https://issues.apache.org/jira/browse/METRON-1731 This is an improvement on handling existing PCAP output directory writing. The PcapJob is extremely permissive in values it accepts for filtering packets, however the fixed/query filter parameters are used for constructing the output directory of the MapReduce job. When the query/field value contains a colon ":", this causes the hdfs Path object to throw a URIException. **Includes:** - token replace colons with underscores in output dir name - consolidate redundant unit test classes for Fixed and Query PCAP filters. - create an output directory formatter class to enable testing in isolation from the rest of pcapjob - consolidates the char escape convention used by fixed and query filter dir naming. Currently handle spaces, periods, apostrophes, and colons. **Testing** This doesn't technically require test data to validate, since it's an error that occurs prior to the job starting, though you're certainly welcome to. Run the following from the CLI and verify no exception is thrown. ``` $METRON_HOME/bin/pcap_query.sh fixed -bp /apps/metron/pcap/input -df "-MM-dd-HH-mm" -st 2018-07-26-00-00 -et 2018-07-26-00-10 -rpf 500 -p "Protocol: ICMP (1)" ``` ## Pull Request Checklist ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: n/a Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mmiklavc/metron output-dir-formatter Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1155.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1155 commit 16ffcc16bef169d9082151b9b2fa381512bc813a Author: Michael Miklavcic Date: 2018-08-08T20:56:10Z Extract output directory formatter, add tests, consolidate redundant pcap filter unit test classes. ---
[jira] [Commented] (METRON-1731) PCAP - Escape colons in output dir names
[ https://issues.apache.org/jira/browse/METRON-1731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573889#comment-16573889 ] ASF GitHub Bot commented on METRON-1731: GitHub user mmiklavc opened a pull request: https://github.com/apache/metron/pull/1155 METRON-1731: PCAP - Escape colons in output dir names ## Contributor Comments https://issues.apache.org/jira/browse/METRON-1731 This is an improvement on handling existing PCAP output directory writing. The PcapJob is extremely permissive in values it accepts for filtering packets, however the fixed/query filter parameters are used for constructing the output directory of the MapReduce job. When the query/field value contains a colon ":", this causes the hdfs Path object to throw a URIException. **Includes:** - token replace colons with underscores in output dir name - consolidate redundant unit test classes for Fixed and Query PCAP filters. - create an output directory formatter class to enable testing in isolation from the rest of pcapjob - consolidates the char escape convention used by fixed and query filter dir naming. Currently handle spaces, periods, apostrophes, and colons. **Testing** This doesn't technically require test data to validate, since it's an error that occurs prior to the job starting, though you're certainly welcome to. Run the following from the CLI and verify no exception is thrown. ``` $METRON_HOME/bin/pcap_query.sh fixed -bp /apps/metron/pcap/input -df "-MM-dd-HH-mm" -st 2018-07-26-00-00 -et 2018-07-26-00-10 -rpf 500 -p "Protocol: ICMP (1)" ``` ## Pull Request Checklist ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: n/a Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mmiklavc/metron output-dir-formatter Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1155.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1155 commit 16ffcc16bef169d9082151b9b2fa381512bc813a Author: Michael Miklavcic Date: 2018-08-08T20:56:10Z Extract output directory formatter, add tests, consolidate redundant pcap filter unit test classes. > PCAP - Escape colons in output dir names > > > Key: METRON-1731 > URL: https://issues.apache.org/jira/browse/METRON-1731 > Project: Metron > Issue Type: Sub-task >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1152: METRON-1730: Update steps to run pycapa on Centos...
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1152#discussion_r208742519
--- Diff: metron-sensors/pycapa/README.md ---
@@ -76,8 +79,83 @@ General notes on the installation of Pycapa.
python setup.py install
```
+### Centos 6
+
+* These instructions can be used directly on CentOS 6 - useful for
developers using the Full Dev Vagrant test box.
+* Older distributions, like CentOS 6, that come with Python 2.6 installed,
should install Python 2.7 within a virtual environment and then run Pycapa from
within the virtual environment.
+
+1. Set up a couple environment variables.
+
+```
+PYCAPA_HOME=/opt/pycapa
+PYTHON27_HOME=/opt/rh/python27/root
+```
+
+1. Install required packages.
+
+```
+for item in epel-release centos-release-scl "@Development tools"
python27 python27-scldevel python27-python-virtualenv libpcap-devel
libselinux-python; do yum install -y $item; done
+```
+
+1. Setup Pycapa directory.
+
+```
+mkdir $PYCAPA_HOME && chmod 755 $PYCAPA_HOME
+```
+
+1. Create the virtualenv.
+
+```
+export LD_LIBRARY_PATH="/opt/rh/python27/root/usr/lib64"
+cd $PYCAPA_HOME
+${PYTHON27_HOME}/usr/bin/virtualenv pycapa-venv
+```
+
+1. Install Librdkafka at your chosen $PREFIX.
+
+```
+export PREFIX=/usr
+wget https://github.com/edenhill/librdkafka/archive/v0.11.5.tar.gz
-O - | tar -xz
--- End diff --
I'm going to set the versions accordingly for these 3 in requirements.txt:
confluent_kafka
pcapy
argparse
```
# pip freeze
You are using pip version 7.1.0, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
confluent-kafka==0.11.5
enum34==1.1.6
futures==3.2.0
pcapy==0.11.4
pycapa==0.1
wheel==0.24.0
```
i.e.
```
confluent-kafka==0.11.5
pcapy==0.11.4
argparse==1.4.0
```
---
[jira] [Commented] (METRON-1730) Update steps to run pycapa on Centos 6
[
https://issues.apache.org/jira/browse/METRON-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573913#comment-16573913
]
ASF GitHub Bot commented on METRON-1730:
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1152#discussion_r208742519
--- Diff: metron-sensors/pycapa/README.md ---
@@ -76,8 +79,83 @@ General notes on the installation of Pycapa.
python setup.py install
```
+### Centos 6
+
+* These instructions can be used directly on CentOS 6 - useful for
developers using the Full Dev Vagrant test box.
+* Older distributions, like CentOS 6, that come with Python 2.6 installed,
should install Python 2.7 within a virtual environment and then run Pycapa from
within the virtual environment.
+
+1. Set up a couple environment variables.
+
+```
+PYCAPA_HOME=/opt/pycapa
+PYTHON27_HOME=/opt/rh/python27/root
+```
+
+1. Install required packages.
+
+```
+for item in epel-release centos-release-scl "@Development tools"
python27 python27-scldevel python27-python-virtualenv libpcap-devel
libselinux-python; do yum install -y $item; done
+```
+
+1. Setup Pycapa directory.
+
+```
+mkdir $PYCAPA_HOME && chmod 755 $PYCAPA_HOME
+```
+
+1. Create the virtualenv.
+
+```
+export LD_LIBRARY_PATH="/opt/rh/python27/root/usr/lib64"
+cd $PYCAPA_HOME
+${PYTHON27_HOME}/usr/bin/virtualenv pycapa-venv
+```
+
+1. Install Librdkafka at your chosen $PREFIX.
+
+```
+export PREFIX=/usr
+wget https://github.com/edenhill/librdkafka/archive/v0.11.5.tar.gz
-O - | tar -xz
--- End diff --
I'm going to set the versions accordingly for these 3 in requirements.txt:
confluent_kafka
pcapy
argparse
```
# pip freeze
You are using pip version 7.1.0, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
confluent-kafka==0.11.5
enum34==1.1.6
futures==3.2.0
pcapy==0.11.4
pycapa==0.1
wheel==0.24.0
```
i.e.
```
confluent-kafka==0.11.5
pcapy==0.11.4
argparse==1.4.0
```
> Update steps to run pycapa on Centos 6
> --
>
> Key: METRON-1730
> URL: https://issues.apache.org/jira/browse/METRON-1730
> Project: Metron
> Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1156: METRON-1702: Reload a running job in the UI
GitHub user merrimanr opened a pull request: https://github.com/apache/metron/pull/1156 METRON-1702: Reload a running job in the UI ## Contributor Comments This PR update the Pcap query panel so that a currently running query for a user is loaded when a page is refreshed. ### Changes Included - Updated pcap-panel.component.ts to query for a running job, and if found, update the current status and start polling - Refactored pcap-panel.component.ts by moving updating status and start polling into separate functions. This should make the class more readable and is needed to update status after querying for a running job. - Updated pcap-filters.component.ts to accept a PcapRequest as input and update filters to match the request made for the currently running job - Added functions to pcap.service.ts to get the currently running job and the pcap request for that job - Fixed a bug in PcapServiceImpl.java where the wrong types were being used for some fields (also adjust unit tests to catch these errors in the future) - Added tests for new code ### Testing This has been tested in full dev. 1. Navigate to the Pcap panel in the Alerts UI 2. Submit a query with filters and wait for progress to be > 0 3. Refresh the page 4. The page should look exactly the same (filters and all) and polling should continue until the job is complete ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/merrimanr/incubator-metron METRON-1702 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1156.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1156 commit 872d1b1ee13e358c18956945d71d3667d19fca8a Author: merrimanr Date: 2018-04-12T14:57:48Z Merge branch 'pcap-front' of https://github.com/simonellistonball/metron into pcaprest Conflicts: metron-interface/metron-alerts/src/app/app.module.ts commit b1b6a7dabea1a1d0d132482c8d97af29c0ac2683 Author: merrimanr Date: 2018-04-13T15:00:15Z
[jira] [Commented] (METRON-1702) Reload a running job in the UI
[ https://issues.apache.org/jira/browse/METRON-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573918#comment-16573918 ] ASF GitHub Bot commented on METRON-1702: GitHub user merrimanr opened a pull request: https://github.com/apache/metron/pull/1156 METRON-1702: Reload a running job in the UI ## Contributor Comments This PR update the Pcap query panel so that a currently running query for a user is loaded when a page is refreshed. ### Changes Included - Updated pcap-panel.component.ts to query for a running job, and if found, update the current status and start polling - Refactored pcap-panel.component.ts by moving updating status and start polling into separate functions. This should make the class more readable and is needed to update status after querying for a running job. - Updated pcap-filters.component.ts to accept a PcapRequest as input and update filters to match the request made for the currently running job - Added functions to pcap.service.ts to get the currently running job and the pcap request for that job - Fixed a bug in PcapServiceImpl.java where the wrong types were being used for some fields (also adjust unit tests to catch these errors in the future) - Added tests for new code ### Testing This has been tested in full dev. 1. Navigate to the Pcap panel in the Alerts UI 2. Submit a query with filters and wait for progress to be > 0 3. Refresh the page 4. The page should look exactly the same (filters and all) and polling should continue until the job is complete ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/merrimanr/incubator-metron METRON-1702 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1156.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1156 commit 872d1b1ee13e358c18956945d71d3667d19fca8a Author: merrimanr Date: 2018-04-12T14:57:48Z Merge bran
[GitHub] metron issue #1142: METRON-1712: PCAP UI Input validation
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1142 This has been merged. Can you close @tiborm? ---
[jira] [Commented] (METRON-1712) PCAP UI - Input validation
[ https://issues.apache.org/jira/browse/METRON-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573923#comment-16573923 ] ASF GitHub Bot commented on METRON-1712: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1142 This has been merged. Can you close @tiborm? > PCAP UI - Input validation > -- > > Key: METRON-1712 > URL: https://issues.apache.org/jira/browse/METRON-1712 > Project: Metron > Issue Type: Sub-task >Reporter: Shane Ardell >Priority: Major > > The following fields need input validation: > IP Source Address: should accept valid IPv4 address formats (v6 is not yet > supported) > IP Source Port: should accept integers from 0 to 65535 > IP Source Address: should accept valid IPv4 address formats (v6 is not yet > supported) > IP Dest Port: should accept integers from 0 to 65535 > Leave unvalidated: > Protocol: number of valid protocols is around 150 and it could change in the > future, validation possible makes more sense on the backend or by a separated > endpoint > Free text filtering: binary regular expression > Broken inputs should be recognized by the UI and not submitted, with > user-facing feedback. The REST API should also fail on queries outside > reasonable parameters. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1152: METRON-1730: Update steps to run pycapa on Centos...
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1152#discussion_r208745344
--- Diff: metron-sensors/pycapa/README.md ---
@@ -76,8 +79,83 @@ General notes on the installation of Pycapa.
python setup.py install
```
+### Centos 6
+
+* These instructions can be used directly on CentOS 6 - useful for
developers using the Full Dev Vagrant test box.
+* Older distributions, like CentOS 6, that come with Python 2.6 installed,
should install Python 2.7 within a virtual environment and then run Pycapa from
within the virtual environment.
+
+1. Set up a couple environment variables.
+
+```
+PYCAPA_HOME=/opt/pycapa
+PYTHON27_HOME=/opt/rh/python27/root
+```
+
+1. Install required packages.
+
+```
+for item in epel-release centos-release-scl "@Development tools"
python27 python27-scldevel python27-python-virtualenv libpcap-devel
libselinux-python; do yum install -y $item; done
+```
+
+1. Setup Pycapa directory.
+
+```
+mkdir $PYCAPA_HOME && chmod 755 $PYCAPA_HOME
+```
+
+1. Create the virtualenv.
+
+```
+export LD_LIBRARY_PATH="/opt/rh/python27/root/usr/lib64"
+cd $PYCAPA_HOME
+${PYTHON27_HOME}/usr/bin/virtualenv pycapa-venv
+```
+
+1. Install Librdkafka at your chosen $PREFIX.
+
+```
+export PREFIX=/usr
+wget https://github.com/edenhill/librdkafka/archive/v0.11.5.tar.gz
-O - | tar -xz
--- End diff --
@justinleet Let me know what you think of the most recent update. That
should set the versions in requirements.txt. When I run it I get the following
output:
```
(pycapa-venv)[root@node1(127.0.0.1 192.168.66.121): /opt/pycapa/pycapa]
# pip install -r requirements.txt
You are using pip version 7.1.0, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting confluent-kafka==0.11.5 (from -r requirements.txt (line 1))
Collecting pcapy==0.11.4 (from -r requirements.txt (line 2))
Collecting argparse==1.4.0 (from -r requirements.txt (line 3))
Using cached
https://files.pythonhosted.org/packages/f2/94/3af39d34be01a24a6e65433d19e107099374224905f1e0cc6bbe1fd22a2f/argparse-1.4.0-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): futures in
/opt/pycapa/pycapa-venv/lib/python2.7/site-packages (from
confluent-kafka==0.11.5->-r requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): enum34 in
/opt/pycapa/pycapa-venv/lib/python2.7/site-packages (from
confluent-kafka==0.11.5->-r requirements.txt (line 1))
Installing collected packages: confluent-kafka, pcapy, argparse
Successfully installed argparse-1.4.0 confluent-kafka-0.11.5 pcapy-0.11.4
(pycapa-venv)[root@node1(127.0.0.1 192.168.66.121):
/opt/pycapa/pycapa-venv/bin]
# pycapa --producer --kafka-topic pcap --interface eth1 --kafka-broker
$BROKERLIST
INFO:root:Connecting to Kafka; {'bootstrap.servers': 'node1:6667',
'group.id': 'TAAAIEXWVROR'}
INFO:root:Starting packet capture
%3|1533038296.388|FAIL|rdkafka#producer-1| [thrd:node1:6667/bootstrap]:
node1:6667/bootstrap: Connect to ipv4#192.168.66.121:6667 failed: Connection
refused
%3|1533038296.388|ERROR|rdkafka#producer-1| [thrd:node1:6667/bootstrap]:
node1:6667/bootstrap: Connect to ipv4#192.168.66.121:6667 failed: Connection
refused
%3|1533038296.388|ERROR|rdkafka#producer-1| [thrd:node1:6667/bootstrap]:
1/1 brokers are down
^CINFO:root:Clean shutdown process started
INFO:root:Waiting for '0' message(s) to flush
INFO:root:'35' packet(s) in, '35' packet(s) out
```
I'm not sure if anyone else gets the FAIL/ERROR messages when they run
pycapa, but it still runs as expected in spite of these errors.
---
[jira] [Commented] (METRON-1730) Update steps to run pycapa on Centos 6
[
https://issues.apache.org/jira/browse/METRON-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573928#comment-16573928
]
ASF GitHub Bot commented on METRON-1730:
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1152#discussion_r208745344
--- Diff: metron-sensors/pycapa/README.md ---
@@ -76,8 +79,83 @@ General notes on the installation of Pycapa.
python setup.py install
```
+### Centos 6
+
+* These instructions can be used directly on CentOS 6 - useful for
developers using the Full Dev Vagrant test box.
+* Older distributions, like CentOS 6, that come with Python 2.6 installed,
should install Python 2.7 within a virtual environment and then run Pycapa from
within the virtual environment.
+
+1. Set up a couple environment variables.
+
+```
+PYCAPA_HOME=/opt/pycapa
+PYTHON27_HOME=/opt/rh/python27/root
+```
+
+1. Install required packages.
+
+```
+for item in epel-release centos-release-scl "@Development tools"
python27 python27-scldevel python27-python-virtualenv libpcap-devel
libselinux-python; do yum install -y $item; done
+```
+
+1. Setup Pycapa directory.
+
+```
+mkdir $PYCAPA_HOME && chmod 755 $PYCAPA_HOME
+```
+
+1. Create the virtualenv.
+
+```
+export LD_LIBRARY_PATH="/opt/rh/python27/root/usr/lib64"
+cd $PYCAPA_HOME
+${PYTHON27_HOME}/usr/bin/virtualenv pycapa-venv
+```
+
+1. Install Librdkafka at your chosen $PREFIX.
+
+```
+export PREFIX=/usr
+wget https://github.com/edenhill/librdkafka/archive/v0.11.5.tar.gz
-O - | tar -xz
--- End diff --
@justinleet Let me know what you think of the most recent update. That
should set the versions in requirements.txt. When I run it I get the following
output:
```
(pycapa-venv)[root@node1(127.0.0.1 192.168.66.121): /opt/pycapa/pycapa]
# pip install -r requirements.txt
You are using pip version 7.1.0, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting confluent-kafka==0.11.5 (from -r requirements.txt (line 1))
Collecting pcapy==0.11.4 (from -r requirements.txt (line 2))
Collecting argparse==1.4.0 (from -r requirements.txt (line 3))
Using cached
https://files.pythonhosted.org/packages/f2/94/3af39d34be01a24a6e65433d19e107099374224905f1e0cc6bbe1fd22a2f/argparse-1.4.0-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): futures in
/opt/pycapa/pycapa-venv/lib/python2.7/site-packages (from
confluent-kafka==0.11.5->-r requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): enum34 in
/opt/pycapa/pycapa-venv/lib/python2.7/site-packages (from
confluent-kafka==0.11.5->-r requirements.txt (line 1))
Installing collected packages: confluent-kafka, pcapy, argparse
Successfully installed argparse-1.4.0 confluent-kafka-0.11.5 pcapy-0.11.4
(pycapa-venv)[root@node1(127.0.0.1 192.168.66.121):
/opt/pycapa/pycapa-venv/bin]
# pycapa --producer --kafka-topic pcap --interface eth1 --kafka-broker
$BROKERLIST
INFO:root:Connecting to Kafka; {'bootstrap.servers': 'node1:6667',
'group.id': 'TAAAIEXWVROR'}
INFO:root:Starting packet capture
%3|1533038296.388|FAIL|rdkafka#producer-1| [thrd:node1:6667/bootstrap]:
node1:6667/bootstrap: Connect to ipv4#192.168.66.121:6667 failed: Connection
refused
%3|1533038296.388|ERROR|rdkafka#producer-1| [thrd:node1:6667/bootstrap]:
node1:6667/bootstrap: Connect to ipv4#192.168.66.121:6667 failed: Connection
refused
%3|1533038296.388|ERROR|rdkafka#producer-1| [thrd:node1:6667/bootstrap]:
1/1 brokers are down
^CINFO:root:Clean shutdown process started
INFO:root:Waiting for '0' message(s) to flush
INFO:root:'35' packet(s) in, '35' packet(s) out
```
I'm not sure if anyone else gets the FAIL/ERROR messages when they run
pycapa, but it still runs as expected in spite of these errors.
> Update steps to run pycapa on Centos 6
> --
>
> Key: METRON-1730
> URL: https://issues.apache.org/jira/browse/METRON-1730
> Project: Metron
> Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (METRON-1702) Reload a running job in the UI
[ https://issues.apache.org/jira/browse/METRON-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman reassigned METRON-1702: - Assignee: Ryan Merriman > Reload a running job in the UI > -- > > Key: METRON-1702 > URL: https://issues.apache.org/jira/browse/METRON-1702 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > When a user reloads the UI, the state of a running pcap job is lost. We need > a way to look up and load a running job when a page is created. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1142: METRON-1712: PCAP UI Input validation
Github user tiborm closed the pull request at: https://github.com/apache/metron/pull/1142 ---
[GitHub] metron issue #1142: METRON-1712: PCAP UI Input validation
Github user tiborm commented on the issue: https://github.com/apache/metron/pull/1142 Thanks for the review! ---
[jira] [Commented] (METRON-1712) PCAP UI - Input validation
[ https://issues.apache.org/jira/browse/METRON-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573999#comment-16573999 ] ASF GitHub Bot commented on METRON-1712: Github user tiborm closed the pull request at: https://github.com/apache/metron/pull/1142 > PCAP UI - Input validation > -- > > Key: METRON-1712 > URL: https://issues.apache.org/jira/browse/METRON-1712 > Project: Metron > Issue Type: Sub-task >Reporter: Shane Ardell >Priority: Major > > The following fields need input validation: > IP Source Address: should accept valid IPv4 address formats (v6 is not yet > supported) > IP Source Port: should accept integers from 0 to 65535 > IP Source Address: should accept valid IPv4 address formats (v6 is not yet > supported) > IP Dest Port: should accept integers from 0 to 65535 > Leave unvalidated: > Protocol: number of valid protocols is around 150 and it could change in the > future, validation possible makes more sense on the backend or by a separated > endpoint > Free text filtering: binary regular expression > Broken inputs should be recognized by the UI and not submitted, with > user-facing feedback. The REST API should also fail on queries outside > reasonable parameters. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1712) PCAP UI - Input validation
[ https://issues.apache.org/jira/browse/METRON-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573998#comment-16573998 ] ASF GitHub Bot commented on METRON-1712: Github user tiborm commented on the issue: https://github.com/apache/metron/pull/1142 Thanks for the review! > PCAP UI - Input validation > -- > > Key: METRON-1712 > URL: https://issues.apache.org/jira/browse/METRON-1712 > Project: Metron > Issue Type: Sub-task >Reporter: Shane Ardell >Priority: Major > > The following fields need input validation: > IP Source Address: should accept valid IPv4 address formats (v6 is not yet > supported) > IP Source Port: should accept integers from 0 to 65535 > IP Source Address: should accept valid IPv4 address formats (v6 is not yet > supported) > IP Dest Port: should accept integers from 0 to 65535 > Leave unvalidated: > Protocol: number of valid protocols is around 150 and it could change in the > future, validation possible makes more sense on the backend or by a separated > endpoint > Free text filtering: binary regular expression > Broken inputs should be recognized by the UI and not submitted, with > user-facing feedback. The REST API should also fail on queries outside > reasonable parameters. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1153: METRON-1725: Add ability to specify YARN queue fo...
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1153#discussion_r208773887
--- Diff:
metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/query/PcapCliTest.java
---
@@ -112,7 +120,24 @@ public void
runs_fixed_pcap_filter_job_with_default_argument_list() throws Excep
return new TypeSafeMatcher>() {
@Override
protected boolean matchesSafely(Map item) {
-return item.entrySet().containsAll(map.entrySet());
+for(K key: map.keySet()) {
+ if (key.equals(PcapOptions.HADOOP_CONF.getKey())) {
+Configuration itemConfiguration = (Configuration)
item.get(PcapOptions.HADOOP_CONF.getKey());
+Map mapConfiguration = (Map)
map.get(PcapOptions.HADOOP_CONF.getKey());
+for(String setting: mapConfiguration.keySet()) {
+ if
(!mapConfiguration.get(setting).equals(itemConfiguration.get(setting, ""))) {
+return false;
+ }
+}
+ } else {
+V itemValue = item.get(key);
+V mapValue = map.get(key);
+if (itemValue != null ? !itemValue.equals(mapValue) : mapValue
== null) {
--- End diff --
Should this be `!(mapValue == null)`?
**Similarly stated:**
```
if (itemValue == null ? mapValue == null : itemValue.equals(mapValue)) {
return true;
}
```
If both values are null, it should return `true`, I would expect.
---
[jira] [Commented] (METRON-1725) Add ability to specify YARN queue for pcap jobs
[
https://issues.apache.org/jira/browse/METRON-1725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574059#comment-16574059
]
ASF GitHub Bot commented on METRON-1725:
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1153#discussion_r208773887
--- Diff:
metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/query/PcapCliTest.java
---
@@ -112,7 +120,24 @@ public void
runs_fixed_pcap_filter_job_with_default_argument_list() throws Excep
return new TypeSafeMatcher>() {
@Override
protected boolean matchesSafely(Map item) {
-return item.entrySet().containsAll(map.entrySet());
+for(K key: map.keySet()) {
+ if (key.equals(PcapOptions.HADOOP_CONF.getKey())) {
+Configuration itemConfiguration = (Configuration)
item.get(PcapOptions.HADOOP_CONF.getKey());
+Map mapConfiguration = (Map)
map.get(PcapOptions.HADOOP_CONF.getKey());
+for(String setting: mapConfiguration.keySet()) {
+ if
(!mapConfiguration.get(setting).equals(itemConfiguration.get(setting, ""))) {
+return false;
+ }
+}
+ } else {
+V itemValue = item.get(key);
+V mapValue = map.get(key);
+if (itemValue != null ? !itemValue.equals(mapValue) : mapValue
== null) {
--- End diff --
Should this be `!(mapValue == null)`?
**Similarly stated:**
```
if (itemValue == null ? mapValue == null : itemValue.equals(mapValue)) {
return true;
}
```
If both values are null, it should return `true`, I would expect.
> Add ability to specify YARN queue for pcap jobs
> ---
>
> Key: METRON-1725
> URL: https://issues.apache.org/jira/browse/METRON-1725
> Project: Metron
> Issue Type: Sub-task
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
>
> There should be a way to configure which YARN queue a pcap query job is
> submitted to. This option should be available for both the CLI and REST.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1153: METRON-1725: Add ability to specify YARN queue for pcap ...
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1153 Thanks for the PR @merrimanr! Any reason we wouldn't expose this option via Ambari while we're at it? We are already setting page size there as well.  ---
[jira] [Commented] (METRON-1725) Add ability to specify YARN queue for pcap jobs
[ https://issues.apache.org/jira/browse/METRON-1725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574061#comment-16574061 ] ASF GitHub Bot commented on METRON-1725: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1153 Thanks for the PR @merrimanr! Any reason we wouldn't expose this option via Ambari while we're at it? We are already setting page size there as well.  > Add ability to specify YARN queue for pcap jobs > --- > > Key: METRON-1725 > URL: https://issues.apache.org/jira/browse/METRON-1725 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > There should be a way to configure which YARN queue a pcap query job is > submitted to. This option should be available for both the CLI and REST. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
