[GitHub] metron issue #1206: METRON-1786: Pcap Topology Status Incorrect
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1206 +1 Verified the fix on full dev. The topology status is seen to be fine after the fix. No errors in ambari agent log. ---
[jira] [Commented] (METRON-1786) Pcap Topology Status Incorrect
[
https://issues.apache.org/jira/browse/METRON-1786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16623292#comment-16623292
]
ASF GitHub Bot commented on METRON-1786:
Github user anandsubbu commented on the issue:
https://github.com/apache/metron/pull/1206
+1
Verified the fix on full dev. The topology status is seen to be fine after
the fix. No errors in ambari agent log.
> Pcap Topology Status Incorrect
> --
>
> Key: METRON-1786
> URL: https://issues.apache.org/jira/browse/METRON-1786
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Mohan
>Priority: Major
>
> After running for some period of time, the pcap topology status in Ambari
> goes to "red" indicating that it is not running, when it actually is running.
> The following exception is thrown in the Ambari Agent logs.
> {code:java}
> INFO 2018-09-20 16:48:08,199 PythonReflectiveExecutor.py:67 - Reflective
> command failed with exception:
> Traceback (most recent call last):
> File
> "/usr/lib/python2.6/site-packages/ambari_agent/PythonReflectiveExecutor.py",
> line 59, in run_file
> imp.load_source('__main__', script)
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.6.0/package/scripts/pcap_master.py",
> line 105, in
> Pcap().execute()
> File
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
> line 375, in execute
> method(env)
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.6.0/package/scripts/pcap_master.py",
> line 93, in status
> commands = PcapCommands(status_params)
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.6.0/package/scripts/pcap_commands.py",
> line 51, in __init__
> self.__pcap_topology = params.metron_pcap_topology
> AttributeError: 'module' object has no attribute 'metron_pcap_topology'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (METRON-1790) Unsubscribe from every observable in the pcap panel UI component
Tamas Fodor created METRON-1790: --- Summary: Unsubscribe from every observable in the pcap panel UI component Key: METRON-1790 URL: https://issues.apache.org/jira/browse/METRON-1790 Project: Metron Issue Type: Improvement Reporter: Tamas Fodor Assignee: Tamas Fodor There are a lot of http requests performed in the pcap panel ui component and we just unsubscribe from some of them when the component is no longer rendered on the screen. It could cause memory consumption issues. Because of the active subscriptions, the garbage collector is not able to remove these objects from the memory, however they're not needed to be there anymore. There's another benefit of unsubscribing from these http calls. If the user leaves the pcap tab but there are pending requests, the unsubscribe method cancels the active xhrs immediately so it won't wait for fulfilment unnecessarily. [https://github.com/apache/metron/blob/master/metron-interface/metron-alerts/src/app/pcap/pcap-panel/pcap-panel.component.ts#L54] I would also refactor that part when we convert an observable to a promise. I would keep it as an observable. By doing this, we would be able to unsubscribe from it as well in the destructor method. Promises are not cancelable. [https://github.com/apache/metron/blob/master/metron-interface/metron-alerts/src/app/pcap/pcap-panel/pcap-panel.component.ts#L70] Resources: [https://angular.io/guide/lifecycle-hooks#ondestroy] This is the place to free resources that won't be garbage collected automatically. Unsubscribe from Observables and DOM events. Stop interval timers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1207: METRON-1695: Expose pcap properties through Ambar...
GitHub user anandsubbu opened a pull request: https://github.com/apache/metron/pull/1207 METRON-1695: Expose pcap properties through Ambari ## Contributor Comments This change makes the PCAP config properties available via the Metron service -> Configs tab in Ambari. It also takes care of updating the `pcap.properties` file with relevant values depending on the deploy environment (single node vs. multi node vs. kerberized cluster). Here's a screenshot of the PCAP Config:  ### Testing Steps * Launch full dev * Metron PCAP service should come up successfully. * Go to Ambari -> Metron services -> Configs -> PCAP tab; change any config parameter; * Restart the required services and the config change take effect. ### Testing Done I ran the following different tests to validate the fix. A. Full Dev * Verified Metron PCAP service comes up successfully * Changed some of the PCAP config values from the Ambari UI, validated that this is reflected in the `pcap.properties` file upon service restart. - **Note**: I ran into [METRON-1786](https://issues.apache.org/jira/browse/METRON-1786) during testing. I was able to resolve the issue by applying the fix in PR #1206 B. Multi-node Cluster * Validated that the `kafka.zk` property in `pcap.properties` is updated with the correct Broker details * Changed a PCAP service-related property (E.g. `kafka_pcap_numpackets`) and validated that the `pcap.properties` file is updated. * Changed a PCAP REST-related property (E.g. `pcap_page_size`) and validated that the `/etc/default/metron` file is updated. C. Kerberized Cluster * Kerberized the multi-node cluster and setup pycapa * Ingested PCAP into the cluster * Ran queries through the PCAP panel and validated that the modified config settings are in effect (E.g. `pcap_page_size` set to 50). ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [NA] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/anandsubbu/incubator-metron METRON-1695-2 Alternativ
[jira] [Commented] (METRON-1695) Expose pcap properties through Ambari
[ https://issues.apache.org/jira/browse/METRON-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16623433#comment-16623433 ] ASF GitHub Bot commented on METRON-1695: GitHub user anandsubbu opened a pull request: https://github.com/apache/metron/pull/1207 METRON-1695: Expose pcap properties through Ambari ## Contributor Comments This change makes the PCAP config properties available via the Metron service -> Configs tab in Ambari. It also takes care of updating the `pcap.properties` file with relevant values depending on the deploy environment (single node vs. multi node vs. kerberized cluster). Here's a screenshot of the PCAP Config:  ### Testing Steps * Launch full dev * Metron PCAP service should come up successfully. * Go to Ambari -> Metron services -> Configs -> PCAP tab; change any config parameter; * Restart the required services and the config change take effect. ### Testing Done I ran the following different tests to validate the fix. A. Full Dev * Verified Metron PCAP service comes up successfully * Changed some of the PCAP config values from the Ambari UI, validated that this is reflected in the `pcap.properties` file upon service restart. - **Note**: I ran into [METRON-1786](https://issues.apache.org/jira/browse/METRON-1786) during testing. I was able to resolve the issue by applying the fix in PR #1206 B. Multi-node Cluster * Validated that the `kafka.zk` property in `pcap.properties` is updated with the correct Broker details * Changed a PCAP service-related property (E.g. `kafka_pcap_numpackets`) and validated that the `pcap.properties` file is updated. * Changed a PCAP REST-related property (E.g. `pcap_page_size`) and validated that the `/etc/default/metron` file is updated. C. Kerberized Cluster * Kerberized the multi-node cluster and setup pycapa * Ingested PCAP into the cluster * Ran queries through the PCAP panel and validated that the modified config settings are in effect (E.g. `pcap_page_size` set to 50). ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [NA] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up
[GitHub] metron pull request #1208: METRON-1790: Unsubscribe from every observable in...
GitHub user ruffle1986 opened a pull request: https://github.com/apache/metron/pull/1208 METRON-1790: Unsubscribe from every observable in the pcap panel UI component ## Contributor Comments Original Jira ticket: https://issues.apache.org/jira/browse/METRON-1790 **Motivation:** There are a lot of http requests performed in the pcap panel ui component and we just unsubscribe from some of them when the component is no longer rendered on the screen. It could cause memory consumption issues. Because of the active subscriptions, the garbage collector is not able to remove these objects from the memory, however they're not needed anymore. There's another benefit of unsubscribing from these http calls. If the user leaves the pcap tab but there are pending requests, the unsubscribe method cancels the active xhrs immediately so it won't wait for fulfilment unnecessarily. I would also refactor that part when we convert an observable to a promise. I would keep it as an observable. By doing this, we would be able to unsubscribe from it as well in the destructor method. Promises are not cancelable. https://github.com/apache/metron/blob/master/metron-interface/metron-alerts/src/app/pcap/pcap-panel/pcap-panel.component.ts#L70 Resources: https://angular.io/guide/lifecycle-hooks#ondestroy > This is the place to free resources that won't be garbage collected automatically. Unsubscribe from Observables and DOM events. Stop interval timers. **Changes** - Collect every subscription in an object and unsubscribe from them by going through the object. - Keep the `getPackets` observable as is. Remove `toPromise()` **Testing** Go to the PCAP tab and start querying or just let the component figure if there are any running jobs (The latter starts an http request immediately when the component is rendered). But before making any http requests, open up the network tab and make sure you throttled the traffic by setting the connection to "Slow 3G" (top right corner) to let the request be in pending state for a while before it fulfils. By this, you get some extra time to change between the PCAP and the Alerts tab. If you change between the tabs and there are any http requests in pending state, they're going to be cancelled and highlighted in red color. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [X] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [X] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [X] Have you included steps or a guide to how the change may be verified and tested manually? - [X] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [X] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
[jira] [Commented] (METRON-1790) Unsubscribe from every observable in the pcap panel UI component
[ https://issues.apache.org/jira/browse/METRON-1790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16623486#comment-16623486 ] ASF GitHub Bot commented on METRON-1790: GitHub user ruffle1986 opened a pull request: https://github.com/apache/metron/pull/1208 METRON-1790: Unsubscribe from every observable in the pcap panel UI component ## Contributor Comments Original Jira ticket: https://issues.apache.org/jira/browse/METRON-1790 **Motivation:** There are a lot of http requests performed in the pcap panel ui component and we just unsubscribe from some of them when the component is no longer rendered on the screen. It could cause memory consumption issues. Because of the active subscriptions, the garbage collector is not able to remove these objects from the memory, however they're not needed anymore. There's another benefit of unsubscribing from these http calls. If the user leaves the pcap tab but there are pending requests, the unsubscribe method cancels the active xhrs immediately so it won't wait for fulfilment unnecessarily. I would also refactor that part when we convert an observable to a promise. I would keep it as an observable. By doing this, we would be able to unsubscribe from it as well in the destructor method. Promises are not cancelable. https://github.com/apache/metron/blob/master/metron-interface/metron-alerts/src/app/pcap/pcap-panel/pcap-panel.component.ts#L70 Resources: https://angular.io/guide/lifecycle-hooks#ondestroy > This is the place to free resources that won't be garbage collected automatically. Unsubscribe from Observables and DOM events. Stop interval timers. **Changes** - Collect every subscription in an object and unsubscribe from them by going through the object. - Keep the `getPackets` observable as is. Remove `toPromise()` **Testing** Go to the PCAP tab and start querying or just let the component figure if there are any running jobs (The latter starts an http request immediately when the component is rendered). But before making any http requests, open up the network tab and make sure you throttled the traffic by setting the connection to "Slow 3G" (top right corner) to let the request be in pending state for a while before it fulfils. By this, you get some extra time to change between the PCAP and the Alerts tab. If you change between the tabs and there are any http requests in pending state, they're going to be cancelled and highlighted in red color. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [X] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [X] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [X] Have you included steps or a guide to how the change may be verified and tested manually? - [X] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [X] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via
[GitHub] metron issue #1185: METRON-1508 In Ubuntu14 Dev Indexing Fails to Write to E...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1185 +1 by inspection ---
[jira] [Commented] (METRON-1508) In Ubuntu14 Dev Indexing Fails to Write to Elasticsearch
[
https://issues.apache.org/jira/browse/METRON-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16623632#comment-16623632
]
ASF GitHub Bot commented on METRON-1508:
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/1185
+1 by inspection
> In Ubuntu14 Dev Indexing Fails to Write to Elasticsearch
>
>
> Key: METRON-1508
> URL: https://issues.apache.org/jira/browse/METRON-1508
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.2
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Minor
>
> When spinning up the Ubuntu 14 development environment, the indexing topology
> fails to write to Elasticsearch. This does not appear to be caused by
> resource constraints or OOM conditions. The indexing topology reports the
> following error.
> {code:java}
> 2018-04-04 15:51:05.707 o.a.s.d.executor Thread-6-indexingBolt-executor[3 3]
> [ERROR]
> org.elasticsearch.client.transport.NoNodeAvailableException: None of the
> configured nodes are available:
> [{#transport#-1}{rZcTXccfSPq4fH4IRLQ0zg}{node1}{127.0.1.1:9300}]
> at
> org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:363)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
> ~[stormjar.jar:?]
> at
> org.apache.metron.elasticsearch.writer.ElasticsearchWriter.write(ElasticsearchWriter.java:92)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.BulkWriterComponent.flush(BulkWriterComponent.java:239)
> [stormjar.jar:?]
> at
> org.apache.metron.writer.BulkWriterComponent.write(BulkWriterComponent.java:217)
> [stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:236)
> [stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__7590$tuple_action_fn__7592.invoke(executor.clj:730)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__7511.invoke(executor.clj:462)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.daemon.executor$fn__7590$fn__7603$fn__7656.invoke(executor.clj:849)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1206: METRON-1786: Pcap Topology Status Incorrect
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1206 ---
[jira] [Commented] (METRON-1786) Pcap Topology Status Incorrect
[
https://issues.apache.org/jira/browse/METRON-1786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16623634#comment-16623634
]
ASF GitHub Bot commented on METRON-1786:
Github user asfgit closed the pull request at:
https://github.com/apache/metron/pull/1206
> Pcap Topology Status Incorrect
> --
>
> Key: METRON-1786
> URL: https://issues.apache.org/jira/browse/METRON-1786
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Mohan
>Priority: Major
>
> After running for some period of time, the pcap topology status in Ambari
> goes to "red" indicating that it is not running, when it actually is running.
> The following exception is thrown in the Ambari Agent logs.
> {code:java}
> INFO 2018-09-20 16:48:08,199 PythonReflectiveExecutor.py:67 - Reflective
> command failed with exception:
> Traceback (most recent call last):
> File
> "/usr/lib/python2.6/site-packages/ambari_agent/PythonReflectiveExecutor.py",
> line 59, in run_file
> imp.load_source('__main__', script)
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.6.0/package/scripts/pcap_master.py",
> line 105, in
> Pcap().execute()
> File
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
> line 375, in execute
> method(env)
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.6.0/package/scripts/pcap_master.py",
> line 93, in status
> commands = PcapCommands(status_params)
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.6.0/package/scripts/pcap_commands.py",
> line 51, in __init__
> self.__pcap_topology = params.metron_pcap_topology
> AttributeError: 'module' object has no attribute 'metron_pcap_topology'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1207: METRON-1695: Expose pcap properties through Ambar...
Github user MohanDV commented on a diff in the pull request: https://github.com/apache/metron/pull/1207#discussion_r219503222 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py --- @@ -387,18 +388,32 @@ bolt_hdfs_rotation_policy_units = config['configurations']['metron-indexing-env']['bolt_hdfs_rotation_policy_units'] bolt_hdfs_rotation_policy_count = config['configurations']['metron-indexing-env']['bolt_hdfs_rotation_policy_count'] -# Pcap +# PCAP metron_pcap_topology = 'pcap' pcap_input_topic = 'pcap' --- End diff -- pcap input topic is also hardcoded in [status_param.py ](https://github.com/MohanDV/metron/blob/a46fda45b46a9f317e2ef7c3755ce6f072ed893b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py#L123) we should get the value from config tab here too. ---
[GitHub] metron pull request #1207: METRON-1695: Expose pcap properties through Ambar...
Github user MohanDV commented on a diff in the pull request: https://github.com/apache/metron/pull/1207#discussion_r219501331 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py --- @@ -387,18 +388,32 @@ bolt_hdfs_rotation_policy_units = config['configurations']['metron-indexing-env']['bolt_hdfs_rotation_policy_units'] bolt_hdfs_rotation_policy_count = config['configurations']['metron-indexing-env']['bolt_hdfs_rotation_policy_count'] -# Pcap +# PCAP metron_pcap_topology = 'pcap' pcap_input_topic = 'pcap' --- End diff -- pcap input topic is also exposed , so shouldn't we be getting the value from config tab ? ---
[jira] [Commented] (METRON-1695) Expose pcap properties through Ambari
[ https://issues.apache.org/jira/browse/METRON-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16623643#comment-16623643 ] ASF GitHub Bot commented on METRON-1695: Github user MohanDV commented on a diff in the pull request: https://github.com/apache/metron/pull/1207#discussion_r219503222 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py --- @@ -387,18 +388,32 @@ bolt_hdfs_rotation_policy_units = config['configurations']['metron-indexing-env']['bolt_hdfs_rotation_policy_units'] bolt_hdfs_rotation_policy_count = config['configurations']['metron-indexing-env']['bolt_hdfs_rotation_policy_count'] -# Pcap +# PCAP metron_pcap_topology = 'pcap' pcap_input_topic = 'pcap' --- End diff -- pcap input topic is also hardcoded in [status_param.py ](https://github.com/MohanDV/metron/blob/a46fda45b46a9f317e2ef7c3755ce6f072ed893b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py#L123) we should get the value from config tab here too. > Expose pcap properties through Ambari > - > > Key: METRON-1695 > URL: https://issues.apache.org/jira/browse/METRON-1695 > Project: Metron > Issue Type: Bug >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > Currently, the $METRON_HOME/config/pcap.properties file is hardcoded with the > defaults. One has to hand edit the file before deploying the PCAP topology. > These properties should be configurable via Ambari. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1695) Expose pcap properties through Ambari
[ https://issues.apache.org/jira/browse/METRON-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16623642#comment-16623642 ] ASF GitHub Bot commented on METRON-1695: Github user MohanDV commented on a diff in the pull request: https://github.com/apache/metron/pull/1207#discussion_r219501331 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py --- @@ -387,18 +388,32 @@ bolt_hdfs_rotation_policy_units = config['configurations']['metron-indexing-env']['bolt_hdfs_rotation_policy_units'] bolt_hdfs_rotation_policy_count = config['configurations']['metron-indexing-env']['bolt_hdfs_rotation_policy_count'] -# Pcap +# PCAP metron_pcap_topology = 'pcap' pcap_input_topic = 'pcap' --- End diff -- pcap input topic is also exposed , so shouldn't we be getting the value from config tab ? > Expose pcap properties through Ambari > - > > Key: METRON-1695 > URL: https://issues.apache.org/jira/browse/METRON-1695 > Project: Metron > Issue Type: Bug >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > Currently, the $METRON_HOME/config/pcap.properties file is hardcoded with the > defaults. One has to hand edit the file before deploying the PCAP topology. > These properties should be configurable via Ambari. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1787) Input Time Constraints for Batch Profiler
[ https://issues.apache.org/jira/browse/METRON-1787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16624017#comment-16624017 ] ASF GitHub Bot commented on METRON-1787: GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/1209 METRON-1787 Input Time Constraints for Batch Profiler This allows the user to define a 'begin' and/or 'end' time constraints to the telemetry that is consumed by the Batch Profiler. See the README for an explanation of these values and how to use them. This is a pull request for the `feature/METRON-1699-create-batch-profiler` feature branch. ## Testing TODO ## Pull Request Checklist - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/metron METRON-1787 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1209.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1209 commit 3a3185f95e2224fd6ca7fa06bf86857a21a8ee0d Author: Nick Allen Date: 2018-09-20T20:45:55Z METRON-1787 Input Time Constraints for Batch Profiler > Input Time Constraints for Batch Profiler > - > > Key: METRON-1787 > URL: https://issues.apache.org/jira/browse/METRON-1787 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > > A user should be able to constrain the input telemetry based on time. The > user might optionally provide a --begin and/or --end command line argument > that defines the time window. > > This was originally [discussed on the mailing > list|https://lists.apache.org/thread.html/39b5636c9ae264611aa8aecc426b649682efaf1bd0755848a90ee6b9@%3Cdev.metron.apache.org%3E]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1209: METRON-1787 Input Time Constraints for Batch Prof...
GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/1209 METRON-1787 Input Time Constraints for Batch Profiler This allows the user to define a 'begin' and/or 'end' time constraints to the telemetry that is consumed by the Batch Profiler. See the README for an explanation of these values and how to use them. This is a pull request for the `feature/METRON-1699-create-batch-profiler` feature branch. ## Testing TODO ## Pull Request Checklist - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/metron METRON-1787 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1209.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1209 commit 3a3185f95e2224fd6ca7fa06bf86857a21a8ee0d Author: Nick Allen Date: 2018-09-20T20:45:55Z METRON-1787 Input Time Constraints for Batch Profiler ---
[GitHub] metron pull request #1185: METRON-1508 In Ubuntu14 Dev Indexing Fails to Wri...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1185 ---
[jira] [Commented] (METRON-1508) In Ubuntu14 Dev Indexing Fails to Write to Elasticsearch
[
https://issues.apache.org/jira/browse/METRON-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16624078#comment-16624078
]
ASF GitHub Bot commented on METRON-1508:
Github user asfgit closed the pull request at:
https://github.com/apache/metron/pull/1185
> In Ubuntu14 Dev Indexing Fails to Write to Elasticsearch
>
>
> Key: METRON-1508
> URL: https://issues.apache.org/jira/browse/METRON-1508
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.2
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Minor
>
> When spinning up the Ubuntu 14 development environment, the indexing topology
> fails to write to Elasticsearch. This does not appear to be caused by
> resource constraints or OOM conditions. The indexing topology reports the
> following error.
> {code:java}
> 2018-04-04 15:51:05.707 o.a.s.d.executor Thread-6-indexingBolt-executor[3 3]
> [ERROR]
> org.elasticsearch.client.transport.NoNodeAvailableException: None of the
> configured nodes are available:
> [{#transport#-1}{rZcTXccfSPq4fH4IRLQ0zg}{node1}{127.0.1.1:9300}]
> at
> org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:363)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
> ~[stormjar.jar:?]
> at
> org.apache.metron.elasticsearch.writer.ElasticsearchWriter.write(ElasticsearchWriter.java:92)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.BulkWriterComponent.flush(BulkWriterComponent.java:239)
> [stormjar.jar:?]
> at
> org.apache.metron.writer.BulkWriterComponent.write(BulkWriterComponent.java:217)
> [stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:236)
> [stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__7590$tuple_action_fn__7592.invoke(executor.clj:730)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__7511.invoke(executor.clj:462)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.daemon.executor$fn__7590$fn__7603$fn__7656.invoke(executor.clj:849)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
