[GitHub] storm pull request: STORM-1674: Addresses network flood from Kafka...
Github user abhishekagarwal87 commented on the pull request: https://github.com/apache/storm/pull/1287#issuecomment-204266196 This should also go to 1.x-branch given that it is bug and can improve the network bandwidth for lower load topologies --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1674) Idle KafkaSpout consumes more bandwidth than needed
[ https://issues.apache.org/jira/browse/STORM-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221227#comment-15221227 ] ASF GitHub Bot commented on STORM-1674: --- Github user abhishekagarwal87 commented on the pull request: https://github.com/apache/storm/pull/1287#issuecomment-204266196 This should also go to 1.x-branch given that it is bug and can improve the network bandwidth for lower load topologies > Idle KafkaSpout consumes more bandwidth than needed > --- > > Key: STORM-1674 > URL: https://issues.apache.org/jira/browse/STORM-1674 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka >Affects Versions: 0.9.3 >Reporter: Robert Hastings > > Discovered 30 megabits of traffic flowing between a set of KafkaSpouts > and our kafka servers even though no Kafka messages were moving. > Using the wireshark kafka dissector, we were able to see that > each FetchRequest had maxWait set to 1 > and minBytes set to 0. When binBytes is set to 0 the kafka server > responds immediately when there are no messages. In turn the KafkaSpout > polls without any delay causing a constant stream of FetchRequest/ > FetchResponse messages. Using a non-KafkaSpout client had a similar > traffic pattern with two key differences > 1) minBytes was 1 > 2) maxWait was 100 > With these FetchRequest parameters and no messages flowing, > the kafka server delays the FetchResponse by 100 ms. This reduces > the network traffic from megabits to the low kilobits. It also > reduced the CPU utilization of our kafka server from 140% to 2%. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-1669) Fix SolrUpdateBolt flush bug
[ https://issues.apache.org/jira/browse/STORM-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] P. Taylor Goetz resolved STORM-1669. Resolution: Fixed Fix Version/s: 2.0.0 1.0.0 Thanks [~vesense]. Merged to master and 1.x-branch. > Fix SolrUpdateBolt flush bug > > > Key: STORM-1669 > URL: https://issues.apache.org/jira/browse/STORM-1669 > Project: Apache Storm > Issue Type: Bug > Components: storm-solr >Affects Versions: 1.0.0, 2.0.0 >Reporter: Xin Wang >Assignee: Xin Wang >Priority: Critical > Fix For: 1.0.0, 2.0.0 > > > SolrUpdateBolt is setting the default tick tuple interval in the prepare() > method, which is not taking effect. > This issue is the same as https://issues.apache.org/jira/browse/STORM-1219 > and https://issues.apache.org/jira/browse/STORM-1654. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1669) Fix SolrUpdateBolt flush bug
[ https://issues.apache.org/jira/browse/STORM-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221188#comment-15221188 ] ASF GitHub Bot commented on STORM-1669: --- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/1284 > Fix SolrUpdateBolt flush bug > > > Key: STORM-1669 > URL: https://issues.apache.org/jira/browse/STORM-1669 > Project: Apache Storm > Issue Type: Bug > Components: storm-solr >Affects Versions: 1.0.0, 2.0.0 >Reporter: Xin Wang >Assignee: Xin Wang >Priority: Critical > > SolrUpdateBolt is setting the default tick tuple interval in the prepare() > method, which is not taking effect. > This issue is the same as https://issues.apache.org/jira/browse/STORM-1219 > and https://issues.apache.org/jira/browse/STORM-1654. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1669 (1.x): Fix SolrUpdateBolt flush bug
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/1284 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1544) Document Debug/Sampling of Topologies
[ https://issues.apache.org/jira/browse/STORM-1544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221169#comment-15221169 ] ASF GitHub Bot commented on STORM-1544: --- Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/1269#issuecomment-204250943 +1 > Document Debug/Sampling of Topologies > - > > Key: STORM-1544 > URL: https://issues.apache.org/jira/browse/STORM-1544 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 1.0.0 >Reporter: P. Taylor Goetz >Assignee: Arun Mahadevan > > Currently the topology/component sampling feature is undocumented, and likely > confusing to users (the UI includes the "Debug" and "Stop Debug", but the > functionality does not provide any indication of what it does, or how to > access the sample logs). > We should document the basic functionality and configuration, as well as how > to extend it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1544] Document Debug/Sampling of Topolo...
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/1269#issuecomment-204250943 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-1649 Optimize Kryo instaces creation in ...
GitHub user satishd opened a pull request: https://github.com/apache/storm/pull/1295 STORM-1649 Optimize Kryo instaces creation in trident windowing This PR is raised as other https://github.com/apache/storm/pull/1273 is merged/closed but it is reverted saying that there was a build error. But [Travis CI build](https://travis-ci.org/apache/storm/builds/119160406) and my local builds are passed. Could not reopen https://github.com/apache/storm/pull/1273 as it is merged/closed. You can merge this pull request into a Git repository by running: $ git pull https://github.com/satishd/storm STORM-1649-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1295.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1295 commit b850847d946ba5ad9809308065352a94bd6287b3 Author: Satish Duggana Date: 2016-03-29T04:32:28Z STORM-1649 kryo serialization in windowing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1669) Fix SolrUpdateBolt flush bug
[ https://issues.apache.org/jira/browse/STORM-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221161#comment-15221161 ] ASF GitHub Bot commented on STORM-1669: --- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/1283 > Fix SolrUpdateBolt flush bug > > > Key: STORM-1669 > URL: https://issues.apache.org/jira/browse/STORM-1669 > Project: Apache Storm > Issue Type: Bug > Components: storm-solr >Affects Versions: 1.0.0, 2.0.0 >Reporter: Xin Wang >Assignee: Xin Wang >Priority: Critical > > SolrUpdateBolt is setting the default tick tuple interval in the prepare() > method, which is not taking effect. > This issue is the same as https://issues.apache.org/jira/browse/STORM-1219 > and https://issues.apache.org/jira/browse/STORM-1654. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1649) Optimize Kryo instaces creation in HBaseWindowsStore
[ https://issues.apache.org/jira/browse/STORM-1649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221164#comment-15221164 ] ASF GitHub Bot commented on STORM-1649: --- GitHub user satishd opened a pull request: https://github.com/apache/storm/pull/1295 STORM-1649 Optimize Kryo instaces creation in trident windowing This PR is raised as other https://github.com/apache/storm/pull/1273 is merged/closed but it is reverted saying that there was a build error. But [Travis CI build](https://travis-ci.org/apache/storm/builds/119160406) and my local builds are passed. Could not reopen https://github.com/apache/storm/pull/1273 as it is merged/closed. You can merge this pull request into a Git repository by running: $ git pull https://github.com/satishd/storm STORM-1649-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1295.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1295 commit b850847d946ba5ad9809308065352a94bd6287b3 Author: Satish Duggana Date: 2016-03-29T04:32:28Z STORM-1649 kryo serialization in windowing > Optimize Kryo instaces creation in HBaseWindowsStore > > > Key: STORM-1649 > URL: https://issues.apache.org/jira/browse/STORM-1649 > Project: Apache Storm > Issue Type: Improvement > Components: trident >Reporter: Satish Duggana >Assignee: Satish Duggana > Fix For: 1.0.0, 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1669: Fix SolrUpdateBolt flush bug
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/1283 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1669) Fix SolrUpdateBolt flush bug
[ https://issues.apache.org/jira/browse/STORM-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221146#comment-15221146 ] ASF GitHub Bot commented on STORM-1669: --- Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/1283#issuecomment-204247353 +1 > Fix SolrUpdateBolt flush bug > > > Key: STORM-1669 > URL: https://issues.apache.org/jira/browse/STORM-1669 > Project: Apache Storm > Issue Type: Bug > Components: storm-solr >Affects Versions: 1.0.0, 2.0.0 >Reporter: Xin Wang >Assignee: Xin Wang >Priority: Critical > > SolrUpdateBolt is setting the default tick tuple interval in the prepare() > method, which is not taking effect. > This issue is the same as https://issues.apache.org/jira/browse/STORM-1219 > and https://issues.apache.org/jira/browse/STORM-1654. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1669: Fix SolrUpdateBolt flush bug
Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/1283#issuecomment-204247353 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1544) Document Debug/Sampling of Topologies
[ https://issues.apache.org/jira/browse/STORM-1544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221138#comment-15221138 ] ASF GitHub Bot commented on STORM-1544: --- Github user arunmahadevan commented on the pull request: https://github.com/apache/storm/pull/1269#issuecomment-204244154 @ptgoetz updated the doc to reflect the new (disabled by default) behavior. Please review. > Document Debug/Sampling of Topologies > - > > Key: STORM-1544 > URL: https://issues.apache.org/jira/browse/STORM-1544 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 1.0.0 >Reporter: P. Taylor Goetz >Assignee: Arun Mahadevan > > Currently the topology/component sampling feature is undocumented, and likely > confusing to users (the UI includes the "Debug" and "Stop Debug", but the > functionality does not provide any indication of what it does, or how to > access the sample logs). > We should document the basic functionality and configuration, as well as how > to extend it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1544] Document Debug/Sampling of Topolo...
Github user arunmahadevan commented on the pull request: https://github.com/apache/storm/pull/1269#issuecomment-204244154 @ptgoetz updated the doc to reflect the new (disabled by default) behavior. Please review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1674) Idle KafkaSpout consumes more bandwidth than needed
[ https://issues.apache.org/jira/browse/STORM-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221116#comment-15221116 ] ASF GitHub Bot commented on STORM-1674: --- Github user moesol commented on the pull request: https://github.com/apache/storm/pull/1287#issuecomment-204239216 KafkaConfig has socketTimeoutMs which I assume maps to socket.timeout.ms. It's passed along to the SimpleConsumer's constructor. I'm not sure yet where rebalance.max.retries etc. would need to get applied. They are not a part of the FetchRequest. My vote is to keep this pull request focused on fixing the issue at hand and have a different pull request to implement the comprehensive list of configurations. Also, I thought I saw some work to create a KafkaSpout that uses the Kafka 0.9 client. Perhaps the comprehensive list of configurations should be coupled with that work. > Idle KafkaSpout consumes more bandwidth than needed > --- > > Key: STORM-1674 > URL: https://issues.apache.org/jira/browse/STORM-1674 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka >Affects Versions: 0.9.3 >Reporter: Robert Hastings > > Discovered 30 megabits of traffic flowing between a set of KafkaSpouts > and our kafka servers even though no Kafka messages were moving. > Using the wireshark kafka dissector, we were able to see that > each FetchRequest had maxWait set to 1 > and minBytes set to 0. When binBytes is set to 0 the kafka server > responds immediately when there are no messages. In turn the KafkaSpout > polls without any delay causing a constant stream of FetchRequest/ > FetchResponse messages. Using a non-KafkaSpout client had a similar > traffic pattern with two key differences > 1) minBytes was 1 > 2) maxWait was 100 > With these FetchRequest parameters and no messages flowing, > the kafka server delays the FetchResponse by 100 ms. This reduces > the network traffic from megabits to the low kilobits. It also > reduced the CPU utilization of our kafka server from 140% to 2%. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1632) Disable event logging by default
[ https://issues.apache.org/jira/browse/STORM-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221115#comment-15221115 ] ASF GitHub Bot commented on STORM-1632: --- Github user arunmahadevan commented on the pull request: https://github.com/apache/storm/pull/1217#issuecomment-204239172 @roshannaik thanks for fixing it in the right way. I have run some quick tests and patch it works as expected. +1 > Disable event logging by default > > > Key: STORM-1632 > URL: https://issues.apache.org/jira/browse/STORM-1632 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Reporter: Roshan Naik >Assignee: Roshan Naik >Priority: Blocker > Attachments: BasicTopology.java > > > EventLogging has performance penalty. For a simple speed of light topology > with a single instances of a spout and a bolt, disabling event logging > delivers a 7% to 9% perf improvement (with acker count =1) > Event logging can be enabled when there is need to do debug, but turned off > by default. > **Update:** with acker=0 the observed impact was much higher... **25%** > faster when event loggers = 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1674: Addresses network flood from Kafka...
Github user moesol commented on the pull request: https://github.com/apache/storm/pull/1287#issuecomment-204239216 KafkaConfig has socketTimeoutMs which I assume maps to socket.timeout.ms. It's passed along to the SimpleConsumer's constructor. I'm not sure yet where rebalance.max.retries etc. would need to get applied. They are not a part of the FetchRequest. My vote is to keep this pull request focused on fixing the issue at hand and have a different pull request to implement the comprehensive list of configurations. Also, I thought I saw some work to create a KafkaSpout that uses the Kafka 0.9 client. Perhaps the comprehensive list of configurations should be coupled with that work. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-1632 Disable event logging by default
Github user arunmahadevan commented on the pull request: https://github.com/apache/storm/pull/1217#issuecomment-204239172 @roshannaik thanks for fixing it in the right way. I have run some quick tests and patch it works as expected. +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-515) Clojure documentation and examples
[ https://issues.apache.org/jira/browse/STORM-515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221108#comment-15221108 ] ASF GitHub Bot commented on STORM-515: -- Github user brymaven commented on the pull request: https://github.com/apache/storm/pull/1262#issuecomment-204238722 Fixed. > Clojure documentation and examples > -- > > Key: STORM-515 > URL: https://issues.apache.org/jira/browse/STORM-515 > Project: Apache Storm > Issue Type: Improvement > Components: documentation, examples >Affects Versions: 0.9.2-incubating >Reporter: Dmitri Sotnikov > Labels: newbie > > Clojure storm-starter example is extremely basic > (https://github.com/apache/storm/blob/master/examples/storm-starter/src/clj/storm/starter/clj/word_count.clj) > and doesn't demonstrate many use cases. > Clojure documentation page > (http://storm.incubator.apache.org/documentation/Clojure-DSL.html) has > unreadable code samples, and glosses over many details in how bolts are > setup, how to initialize parametarized bolts, where the requires such as > execute come from, and so on. > It would be nice to see clear documentation with properly documented examples > for each of the use cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-515: Clojure documentation and examples
Github user brymaven commented on the pull request: https://github.com/apache/storm/pull/1262#issuecomment-204238722 Fixed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1673) log4j2/worker.xml refers old package of LoggerMetricsConsumer
[ https://issues.apache.org/jira/browse/STORM-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221099#comment-15221099 ] ASF GitHub Bot commented on STORM-1673: --- Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/1293#issuecomment-204235510 @knusbaum Hi Kyle, I already submit pull requests #1291 and #1292. Would you please close your pull requests and set assignee to me? Thanks! > log4j2/worker.xml refers old package of LoggerMetricsConsumer > - > > Key: STORM-1673 > URL: https://issues.apache.org/jira/browse/STORM-1673 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.0, 2.0.0 >Reporter: Jungtaek Lim >Assignee: Kyle Nusbaum > > We changed package path from 'backtype.storm' to 'org.apache.storm'. Source > codes seem to moved properly, but missed log4j2 configuration, so metric log > is logged into worker log file, not metrics file. > It should be simple patch so I'd like to include this as 1.0.0. If we don't > want to include any bugfixes I'm OK to remove epic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1673: log4j2/worker.xml refers old packa...
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/1293#issuecomment-204235510 @knusbaum Hi Kyle, I already submit pull requests #1291 and #1292. Would you please close your pull requests and set assignee to me? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (STORM-1674) Idle KafkaSpout consumes more bandwidth than needed
Robert Hastings created STORM-1674: -- Summary: Idle KafkaSpout consumes more bandwidth than needed Key: STORM-1674 URL: https://issues.apache.org/jira/browse/STORM-1674 Project: Apache Storm Issue Type: Bug Components: storm-kafka Affects Versions: 0.9.3 Reporter: Robert Hastings Discovered 30 megabits of traffic flowing between a set of KafkaSpouts and our kafka servers even though no Kafka messages were moving. Using the wireshark kafka dissector, we were able to see that each FetchRequest had maxWait set to 1 and minBytes set to 0. When binBytes is set to 0 the kafka server responds immediately when there are no messages. In turn the KafkaSpout polls without any delay causing a constant stream of FetchRequest/ FetchResponse messages. Using a non-KafkaSpout client had a similar traffic pattern with two key differences 1) minBytes was 1 2) maxWait was 100 With these FetchRequest parameters and no messages flowing, the kafka server delays the FetchResponse by 100 ms. This reduces the network traffic from megabits to the low kilobits. It also reduced the CPU utilization of our kafka server from 140% to 2%. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (STORM-1673) log4j2/worker.xml refers old package of LoggerMetricsConsumer
[ https://issues.apache.org/jira/browse/STORM-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kyle Nusbaum reassigned STORM-1673: --- Assignee: Kyle Nusbaum (was: Jungtaek Lim) > log4j2/worker.xml refers old package of LoggerMetricsConsumer > - > > Key: STORM-1673 > URL: https://issues.apache.org/jira/browse/STORM-1673 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.0, 2.0.0 >Reporter: Jungtaek Lim >Assignee: Kyle Nusbaum > > We changed package path from 'backtype.storm' to 'org.apache.storm'. Source > codes seem to moved properly, but missed log4j2 configuration, so metric log > is logged into worker log file, not metrics file. > It should be simple patch so I'd like to include this as 1.0.0. If we don't > want to include any bugfixes I'm OK to remove epic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1673) log4j2/worker.xml refers old package of LoggerMetricsConsumer
[ https://issues.apache.org/jira/browse/STORM-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221096#comment-15221096 ] ASF GitHub Bot commented on STORM-1673: --- GitHub user knusbaum opened a pull request: https://github.com/apache/storm/pull/1294 STORM-1673: log4j2/worker.xml refers old package of LoggerMetricsConsumer You can merge this pull request into a Git repository by running: $ git pull https://github.com/knusbaum/incubator-storm STORM-1673-master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1294.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1294 commit 33507bd886b19d3cf1abe5063f0c3a0f4850f287 Author: Kyle Nusbaum Date: 2016-04-01T03:28:14Z Updating log4j loggers for org.apache.storm > log4j2/worker.xml refers old package of LoggerMetricsConsumer > - > > Key: STORM-1673 > URL: https://issues.apache.org/jira/browse/STORM-1673 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.0, 2.0.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim > > We changed package path from 'backtype.storm' to 'org.apache.storm'. Source > codes seem to moved properly, but missed log4j2 configuration, so metric log > is logged into worker log file, not metrics file. > It should be simple patch so I'd like to include this as 1.0.0. If we don't > want to include any bugfixes I'm OK to remove epic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1673) log4j2/worker.xml refers old package of LoggerMetricsConsumer
[ https://issues.apache.org/jira/browse/STORM-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221095#comment-15221095 ] ASF GitHub Bot commented on STORM-1673: --- GitHub user knusbaum opened a pull request: https://github.com/apache/storm/pull/1293 STORM-1673: log4j2/worker.xml refers old package of LoggerMetricsConsumer You can merge this pull request into a Git repository by running: $ git pull https://github.com/knusbaum/incubator-storm STORM-1673 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1293.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1293 commit 94b5ca1cd56da9e705f6c4db2c4f80523c9fc554 Author: Kyle Nusbaum Date: 2016-04-01T03:28:14Z Updating log4j loggers for org.apache.storm > log4j2/worker.xml refers old package of LoggerMetricsConsumer > - > > Key: STORM-1673 > URL: https://issues.apache.org/jira/browse/STORM-1673 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.0, 2.0.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim > > We changed package path from 'backtype.storm' to 'org.apache.storm'. Source > codes seem to moved properly, but missed log4j2 configuration, so metric log > is logged into worker log file, not metrics file. > It should be simple patch so I'd like to include this as 1.0.0. If we don't > want to include any bugfixes I'm OK to remove epic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1673: log4j2/worker.xml refers old packa...
GitHub user knusbaum opened a pull request: https://github.com/apache/storm/pull/1293 STORM-1673: log4j2/worker.xml refers old package of LoggerMetricsConsumer You can merge this pull request into a Git repository by running: $ git pull https://github.com/knusbaum/incubator-storm STORM-1673 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1293.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1293 commit 94b5ca1cd56da9e705f6c4db2c4f80523c9fc554 Author: Kyle Nusbaum Date: 2016-04-01T03:28:14Z Updating log4j loggers for org.apache.storm --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-1673: log4j2/worker.xml refers old packa...
GitHub user knusbaum opened a pull request: https://github.com/apache/storm/pull/1294 STORM-1673: log4j2/worker.xml refers old package of LoggerMetricsConsumer You can merge this pull request into a Git repository by running: $ git pull https://github.com/knusbaum/incubator-storm STORM-1673-master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1294.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1294 commit 33507bd886b19d3cf1abe5063f0c3a0f4850f287 Author: Kyle Nusbaum Date: 2016-04-01T03:28:14Z Updating log4j loggers for org.apache.storm --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1673) log4j2/worker.xml refers old package of LoggerMetricsConsumer
[ https://issues.apache.org/jira/browse/STORM-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221091#comment-15221091 ] ASF GitHub Bot commented on STORM-1673: --- GitHub user HeartSaVioR opened a pull request: https://github.com/apache/storm/pull/1292 STORM-1673 log4j2/worker.xml refers old package of LoggerMetricsConsumer * fix missing package changes You can merge this pull request into a Git repository by running: $ git pull https://github.com/HeartSaVioR/storm STORM-1673-1.x Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1292.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1292 commit dd3d0b2000c41b587cb22ecf16ddfd971c0c1d00 Author: Jungtaek Lim Date: 2016-04-01T03:25:48Z STORM-1673 log4j2/worker.xml refers old package of LoggerMetricsConsumer * fix missing package changes > log4j2/worker.xml refers old package of LoggerMetricsConsumer > - > > Key: STORM-1673 > URL: https://issues.apache.org/jira/browse/STORM-1673 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.0, 2.0.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim > > We changed package path from 'backtype.storm' to 'org.apache.storm'. Source > codes seem to moved properly, but missed log4j2 configuration, so metric log > is logged into worker log file, not metrics file. > It should be simple patch so I'd like to include this as 1.0.0. If we don't > want to include any bugfixes I'm OK to remove epic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1673) log4j2/worker.xml refers old package of LoggerMetricsConsumer
[ https://issues.apache.org/jira/browse/STORM-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221090#comment-15221090 ] ASF GitHub Bot commented on STORM-1673: --- GitHub user HeartSaVioR opened a pull request: https://github.com/apache/storm/pull/1291 STORM-1673 log4j2/worker.xml refers old package of LoggerMetricsConsumer * fix missing package changes You can merge this pull request into a Git repository by running: $ git pull https://github.com/HeartSaVioR/storm STORM-1673 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1291.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1291 commit 96da31ed0068dde3b4309ee417382be5084ce941 Author: Jungtaek Lim Date: 2016-04-01T03:22:38Z STORM-1673 log4j2/worker.xml refers old package of LoggerMetricsConsumer * fix missing package changes > log4j2/worker.xml refers old package of LoggerMetricsConsumer > - > > Key: STORM-1673 > URL: https://issues.apache.org/jira/browse/STORM-1673 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.0, 2.0.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim > > We changed package path from 'backtype.storm' to 'org.apache.storm'. Source > codes seem to moved properly, but missed log4j2 configuration, so metric log > is logged into worker log file, not metrics file. > It should be simple patch so I'd like to include this as 1.0.0. If we don't > want to include any bugfixes I'm OK to remove epic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1673 log4j2/worker.xml refers old packag...
GitHub user HeartSaVioR opened a pull request: https://github.com/apache/storm/pull/1291 STORM-1673 log4j2/worker.xml refers old package of LoggerMetricsConsumer * fix missing package changes You can merge this pull request into a Git repository by running: $ git pull https://github.com/HeartSaVioR/storm STORM-1673 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1291.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1291 commit 96da31ed0068dde3b4309ee417382be5084ce941 Author: Jungtaek Lim Date: 2016-04-01T03:22:38Z STORM-1673 log4j2/worker.xml refers old package of LoggerMetricsConsumer * fix missing package changes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-1673 log4j2/worker.xml refers old packag...
GitHub user HeartSaVioR opened a pull request: https://github.com/apache/storm/pull/1292 STORM-1673 log4j2/worker.xml refers old package of LoggerMetricsConsumer * fix missing package changes You can merge this pull request into a Git repository by running: $ git pull https://github.com/HeartSaVioR/storm STORM-1673-1.x Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1292.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1292 commit dd3d0b2000c41b587cb22ecf16ddfd971c0c1d00 Author: Jungtaek Lim Date: 2016-04-01T03:25:48Z STORM-1673 log4j2/worker.xml refers old package of LoggerMetricsConsumer * fix missing package changes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1669) Fix SolrUpdateBolt flush bug
[ https://issues.apache.org/jira/browse/STORM-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221089#comment-15221089 ] ASF GitHub Bot commented on STORM-1669: --- Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/1284#issuecomment-204229422 +1 > Fix SolrUpdateBolt flush bug > > > Key: STORM-1669 > URL: https://issues.apache.org/jira/browse/STORM-1669 > Project: Apache Storm > Issue Type: Bug > Components: storm-solr >Affects Versions: 1.0.0, 2.0.0 >Reporter: Xin Wang >Assignee: Xin Wang >Priority: Critical > > SolrUpdateBolt is setting the default tick tuple interval in the prepare() > method, which is not taking effect. > This issue is the same as https://issues.apache.org/jira/browse/STORM-1219 > and https://issues.apache.org/jira/browse/STORM-1654. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1669 (1.x): Fix SolrUpdateBolt flush bug
Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/1284#issuecomment-204229422 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1661) Introduce a config to turn off blobstore acl validation in insecure mode
[ https://issues.apache.org/jira/browse/STORM-1661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221087#comment-15221087 ] ASF GitHub Bot commented on STORM-1661: --- Github user redsanket commented on the pull request: https://github.com/apache/storm/pull/1271#issuecomment-204228944 @revans2 Added the config > Introduce a config to turn off blobstore acl validation in insecure mode > > > Key: STORM-1661 > URL: https://issues.apache.org/jira/browse/STORM-1661 > Project: Apache Storm > Issue Type: Story >Reporter: Sanket Reddy >Assignee: Sanket Reddy >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1661] Introduce ACL Validation config f...
Github user redsanket commented on the pull request: https://github.com/apache/storm/pull/1271#issuecomment-204228944 @revans2 Added the config --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [YSTORM-1661] Added acl validation config 1.x ...
GitHub user redsanket opened a pull request: https://github.com/apache/storm/pull/1290 [YSTORM-1661] Added acl validation config 1.x Branch PR You can merge this pull request into a Git repository by running: $ git pull https://github.com/redsanket/storm STORM-1661-1.x Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1290.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1290 commit 63f44788c3ca4e6eae2ed134d3d31364757c9fec Author: Sanket Date: 2016-03-30T16:36:27Z added acl validation config --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Reopened] (STORM-1581) Repair github links in the storm documentation
[ https://issues.apache.org/jira/browse/STORM-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kyle Nusbaum reopened STORM-1581: - Assignee: Kyle Nusbaum (was: Abhishek Agarwal) > Repair github links in the storm documentation > -- > > Key: STORM-1581 > URL: https://issues.apache.org/jira/browse/STORM-1581 > Project: Apache Storm > Issue Type: Bug > Components: documentation >Affects Versions: 1.0.0 >Reporter: Abhishek Agarwal >Assignee: Kyle Nusbaum > Fix For: 1.0.0 > > > Java classes have been migrated to org.apache.storm and the github links are > now broken in the documentation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-1581) Repair github links in the storm documentation
[ https://issues.apache.org/jira/browse/STORM-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kyle Nusbaum resolved STORM-1581. - Resolution: Fixed > Repair github links in the storm documentation > -- > > Key: STORM-1581 > URL: https://issues.apache.org/jira/browse/STORM-1581 > Project: Apache Storm > Issue Type: Bug > Components: documentation >Affects Versions: 1.0.0 >Reporter: Abhishek Agarwal >Assignee: Kyle Nusbaum > Fix For: 1.0.0 > > > Java classes have been migrated to org.apache.storm and the github links are > now broken in the documentation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1668) Flux silently fails while setting a non-existent property.
[ https://issues.apache.org/jira/browse/STORM-1668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221075#comment-15221075 ] ASF GitHub Bot commented on STORM-1668: --- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/1281 > Flux silently fails while setting a non-existent property. > -- > > Key: STORM-1668 > URL: https://issues.apache.org/jira/browse/STORM-1668 > Project: Apache Storm > Issue Type: Bug > Components: Flux >Affects Versions: 1.0.0, 2.0.0 >Reporter: Priyank Shah >Assignee: P. Taylor Goetz > Fix For: 1.0.0 > > > Currently, if a yaml file has a property with a name that does not exist on > the java topology component object then flux silently fails by logging a > message and does not throw an exception. This needs to be changed so that > flux throws the exception failing topology submission so that user can take > corrective action. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-1668) Flux silently fails while setting a non-existent property.
[ https://issues.apache.org/jira/browse/STORM-1668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kyle Nusbaum resolved STORM-1668. - Resolution: Fixed > Flux silently fails while setting a non-existent property. > -- > > Key: STORM-1668 > URL: https://issues.apache.org/jira/browse/STORM-1668 > Project: Apache Storm > Issue Type: Bug > Components: Flux >Affects Versions: 1.0.0, 2.0.0 >Reporter: Priyank Shah >Assignee: P. Taylor Goetz > Fix For: 1.0.0 > > > Currently, if a yaml file has a property with a name that does not exist on > the java topology component object then flux silently fails by logging a > message and does not throw an exception. This needs to be changed so that > flux throws the exception failing topology submission so that user can take > corrective action. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1668: Fix silent failing of flux for set...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/1281 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1668) Flux silently fails while setting a non-existent property.
[ https://issues.apache.org/jira/browse/STORM-1668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221072#comment-15221072 ] ASF GitHub Bot commented on STORM-1668: --- Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/1281#issuecomment-204227208 +1 > Flux silently fails while setting a non-existent property. > -- > > Key: STORM-1668 > URL: https://issues.apache.org/jira/browse/STORM-1668 > Project: Apache Storm > Issue Type: Bug > Components: Flux >Affects Versions: 1.0.0, 2.0.0 >Reporter: Priyank Shah >Assignee: P. Taylor Goetz > Fix For: 1.0.0 > > > Currently, if a yaml file has a property with a name that does not exist on > the java topology component object then flux silently fails by logging a > message and does not throw an exception. This needs to be changed so that > flux throws the exception failing topology submission so that user can take > corrective action. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1668: Fix silent failing of flux for set...
Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/1281#issuecomment-204227208 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1668) Flux silently fails while setting a non-existent property.
[ https://issues.apache.org/jira/browse/STORM-1668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221071#comment-15221071 ] ASF GitHub Bot commented on STORM-1668: --- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/1282 > Flux silently fails while setting a non-existent property. > -- > > Key: STORM-1668 > URL: https://issues.apache.org/jira/browse/STORM-1668 > Project: Apache Storm > Issue Type: Bug > Components: Flux >Affects Versions: 1.0.0, 2.0.0 >Reporter: Priyank Shah >Assignee: P. Taylor Goetz > Fix For: 1.0.0 > > > Currently, if a yaml file has a property with a name that does not exist on > the java topology component object then flux silently fails by logging a > message and does not throw an exception. This needs to be changed so that > flux throws the exception failing topology submission so that user can take > corrective action. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-1673) log4j2/worker.xml refers old package of LoggerMetricsConsumer
Jungtaek Lim created STORM-1673: --- Summary: log4j2/worker.xml refers old package of LoggerMetricsConsumer Key: STORM-1673 URL: https://issues.apache.org/jira/browse/STORM-1673 Project: Apache Storm Issue Type: Bug Components: storm-core Affects Versions: 1.0.0, 2.0.0 Reporter: Jungtaek Lim Assignee: Jungtaek Lim We changed package path from 'backtype.storm' to 'org.apache.storm'. Source codes seem to moved properly, but missed log4j2 configuration, so metric log is logged into worker log file, not metrics file. It should be simple patch so I'd like to include this as 1.0.0. If we don't want to include any bugfixes I'm OK to remove epic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1668: Fix silent failing of flux for set...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/1282 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1649) Optimize Kryo instaces creation in HBaseWindowsStore
[ https://issues.apache.org/jira/browse/STORM-1649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221066#comment-15221066 ] ASF GitHub Bot commented on STORM-1649: --- Github user satishd closed the pull request at: https://github.com/apache/storm/pull/1289 > Optimize Kryo instaces creation in HBaseWindowsStore > > > Key: STORM-1649 > URL: https://issues.apache.org/jira/browse/STORM-1649 > Project: Apache Storm > Issue Type: Improvement > Components: trident >Reporter: Satish Duggana >Assignee: Satish Duggana > Fix For: 1.0.0, 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1649 kryo serialization in trident windo...
Github user satishd closed the pull request at: https://github.com/apache/storm/pull/1289 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1649) Optimize Kryo instaces creation in HBaseWindowsStore
[ https://issues.apache.org/jira/browse/STORM-1649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221065#comment-15221065 ] ASF GitHub Bot commented on STORM-1649: --- GitHub user satishd opened a pull request: https://github.com/apache/storm/pull/1289 STORM-1649 kryo serialization in trident windowing You can merge this pull request into a Git repository by running: $ git pull https://github.com/satishd/storm STORM-1649-m Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1289.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1289 commit 690617cf7f19b4345fa6a9834317e5a918c39807 Author: Satish Duggana Date: 2016-03-29T04:32:28Z STORM-1649 kryo serialization in windowing > Optimize Kryo instaces creation in HBaseWindowsStore > > > Key: STORM-1649 > URL: https://issues.apache.org/jira/browse/STORM-1649 > Project: Apache Storm > Issue Type: Improvement > Components: trident >Reporter: Satish Duggana >Assignee: Satish Duggana > Fix For: 1.0.0, 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1649 kryo serialization in trident windo...
GitHub user satishd opened a pull request: https://github.com/apache/storm/pull/1289 STORM-1649 kryo serialization in trident windowing You can merge this pull request into a Git repository by running: $ git pull https://github.com/satishd/storm STORM-1649-m Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1289.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1289 commit 690617cf7f19b4345fa6a9834317e5a918c39807 Author: Satish Duggana Date: 2016-03-29T04:32:28Z STORM-1649 kryo serialization in windowing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1632) Disable event logging by default
[ https://issues.apache.org/jira/browse/STORM-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221063#comment-15221063 ] ASF GitHub Bot commented on STORM-1632: --- Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/1217#issuecomment-204225005 @ptgoetz +1 for quick merge > Disable event logging by default > > > Key: STORM-1632 > URL: https://issues.apache.org/jira/browse/STORM-1632 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Reporter: Roshan Naik >Assignee: Roshan Naik >Priority: Blocker > Attachments: BasicTopology.java > > > EventLogging has performance penalty. For a simple speed of light topology > with a single instances of a spout and a bolt, disabling event logging > delivers a 7% to 9% perf improvement (with acker count =1) > Event logging can be enabled when there is need to do debug, but turned off > by default. > **Update:** with acker=0 the observed impact was much higher... **25%** > faster when event loggers = 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1632 Disable event logging by default
Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/1217#issuecomment-204225005 @ptgoetz +1 for quick merge --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1632) Disable event logging by default
[ https://issues.apache.org/jira/browse/STORM-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221061#comment-15221061 ] ASF GitHub Bot commented on STORM-1632: --- Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/1217#issuecomment-204224677 Thanks for the last-minute effort @roshannaik. +1 I typically like to wait for 24 hrs. after the last commit on a PR to merge, even though it's not required by our bylaws. I intend to merge this earlier in the interest of getting the 1.0 release out. If there are any objections after the fact, there will plenty of time to cancel the VOTE, revert, etc. > Disable event logging by default > > > Key: STORM-1632 > URL: https://issues.apache.org/jira/browse/STORM-1632 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Reporter: Roshan Naik >Assignee: Roshan Naik >Priority: Blocker > Attachments: BasicTopology.java > > > EventLogging has performance penalty. For a simple speed of light topology > with a single instances of a spout and a bolt, disabling event logging > delivers a 7% to 9% perf improvement (with acker count =1) > Event logging can be enabled when there is need to do debug, but turned off > by default. > **Update:** with acker=0 the observed impact was much higher... **25%** > faster when event loggers = 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1632 Disable event logging by default
Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/1217#issuecomment-204224677 Thanks for the last-minute effort @roshannaik. +1 I typically like to wait for 24 hrs. after the last commit on a PR to merge, even though it's not required by our bylaws. I intend to merge this earlier in the interest of getting the 1.0 release out. If there are any objections after the fact, there will plenty of time to cancel the VOTE, revert, etc. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1668) Flux silently fails while setting a non-existent property.
[ https://issues.apache.org/jira/browse/STORM-1668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221048#comment-15221048 ] ASF GitHub Bot commented on STORM-1668: --- Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/1282#issuecomment-204221012 +1 > Flux silently fails while setting a non-existent property. > -- > > Key: STORM-1668 > URL: https://issues.apache.org/jira/browse/STORM-1668 > Project: Apache Storm > Issue Type: Bug > Components: Flux >Affects Versions: 1.0.0, 2.0.0 >Reporter: Priyank Shah >Assignee: P. Taylor Goetz > Fix For: 1.0.0 > > > Currently, if a yaml file has a property with a name that does not exist on > the java topology component object then flux silently fails by logging a > message and does not throw an exception. This needs to be changed so that > flux throws the exception failing topology submission so that user can take > corrective action. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1668: Fix silent failing of flux for set...
Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/1282#issuecomment-204221012 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-1632 Disable event logging by default
Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/1217#issuecomment-204220646 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1632) Disable event logging by default
[ https://issues.apache.org/jira/browse/STORM-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221045#comment-15221045 ] ASF GitHub Bot commented on STORM-1632: --- Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/1217#issuecomment-204220646 +1 > Disable event logging by default > > > Key: STORM-1632 > URL: https://issues.apache.org/jira/browse/STORM-1632 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Reporter: Roshan Naik >Assignee: Roshan Naik >Priority: Blocker > Attachments: BasicTopology.java > > > EventLogging has performance penalty. For a simple speed of light topology > with a single instances of a spout and a bolt, disabling event logging > delivers a 7% to 9% perf improvement (with acker count =1) > Event logging can be enabled when there is need to do debug, but turned off > by default. > **Update:** with acker=0 the observed impact was much higher... **25%** > faster when event loggers = 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1667) Log the IO Exception when deleting worker pid dir
[ https://issues.apache.org/jira/browse/STORM-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221041#comment-15221041 ] ASF GitHub Bot commented on STORM-1667: --- Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/1280#issuecomment-204220497 +1. Will merge tomorrow after 24hr period has passed. > Log the IO Exception when deleting worker pid dir > - > > Key: STORM-1667 > URL: https://issues.apache.org/jira/browse/STORM-1667 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.0 >Reporter: Zhuo Liu >Assignee: Zhuo Liu > Fix For: 1.0.0 > > > There is a race condition since shutdown-worker is called by both > sync-processes and synchronize-supervisor in supervisor. One pid directory > might have been already deleted when rmr-as-user tries to delete it. > Therefore, we should not let the FileNotFoundException get thrown and crash > the supervisor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1667] Log the IO exception when deletin...
Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/1280#issuecomment-204220497 +1. Will merge tomorrow after 24hr period has passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Closed] (STORM-1581) Repair github links in the storm documentation
[ https://issues.apache.org/jira/browse/STORM-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kyle Nusbaum closed STORM-1581. --- Resolution: Fixed Fix Version/s: 1.0.0 Fixed when [~revans2] moved the docs to SVN. > Repair github links in the storm documentation > -- > > Key: STORM-1581 > URL: https://issues.apache.org/jira/browse/STORM-1581 > Project: Apache Storm > Issue Type: Bug > Components: documentation >Affects Versions: 1.0.0 >Reporter: Abhishek Agarwal >Assignee: Abhishek Agarwal > Fix For: 1.0.0 > > > Java classes have been migrated to org.apache.storm and the github links are > now broken in the documentation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: Addresses network flood from KafkaSpout to kaf...
Github user sachingsachin commented on the pull request: https://github.com/apache/storm/pull/1287#issuecomment-204216287 Very useful observation @moesol Comparing the [KafkaConfig.java](https://github.com/apache/storm/blob/a4f9f8bc5b4ca85de487a0a868e519ddcb94e852/external/storm-kafka/src/jvm/org/apache/storm/kafka/KafkaConfig.java) and [Kafka Configuration](https://kafka.apache.org/08/configuration.html), it seems several other useful configurations like socket.timeout.ms, rebalance.max.retries etc. are also absent from KafkaConfig.java We should probably add most (if not all) of these options to our spout-configuration. If we agree that it's useful, then we can add the remaining configuration options to this pull request OR I can submit a new one for the same. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Addresses network flood from KafkaSpout to kaf...
Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/1287#issuecomment-204207397 Please file a [Jira](http://issues.apache.org/jira/) for this and update the title of the PR. I think it should be considered a bug, as this could potentially be detrimental to the network bandwidth. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1649) Optimize Kryo instaces creation in HBaseWindowsStore
[ https://issues.apache.org/jira/browse/STORM-1649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220990#comment-15220990 ] ASF GitHub Bot commented on STORM-1649: --- Github user satishd commented on the pull request: https://github.com/apache/storm/pull/1273#issuecomment-204207233 @ptgoetz TravisCI build is passed for this PR [here](https://travis-ci.org/apache/storm/builds/119160406). I ran locally and I did not face any issues. > Optimize Kryo instaces creation in HBaseWindowsStore > > > Key: STORM-1649 > URL: https://issues.apache.org/jira/browse/STORM-1649 > Project: Apache Storm > Issue Type: Improvement > Components: trident >Reporter: Satish Duggana >Assignee: Satish Duggana > Fix For: 1.0.0, 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1649 Optimize Kryo instaces creation in ...
Github user satishd commented on the pull request: https://github.com/apache/storm/pull/1273#issuecomment-204207233 @ptgoetz TravisCI build is passed for this PR [here](https://travis-ci.org/apache/storm/builds/119160406). I ran locally and I did not face any issues. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-515) Clojure documentation and examples
[
https://issues.apache.org/jira/browse/STORM-515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220983#comment-15220983
]
ASF GitHub Bot commented on STORM-515:
--
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/1262#discussion_r58153392
--- Diff:
examples/storm-starter/src/clj/org/apache/storm/starter/clj/bolts.clj ---
@@ -0,0 +1,78 @@
+;; Licensed to the Apache Software Foundation (ASF) under one
+;; or more contributor license agreements. See the NOTICE file
+;; distributed with this work for additional information
+;; regarding copyright ownership. The ASF licenses this file
+;; to you under the Apache License, Version 2.0 (the
+;; "License"); you may not use this file except in compliance
+;; with the License. You may obtain a copy of the License at
+;;
+;; http://www.apache.org/licenses/LICENSE-2.0
+;;
+;; Unless required by applicable law or agreed to in writing, software
+;; distributed under the License is distributed on an "AS IS" BASIS,
+;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+;; See the License for the specific language governing permissions and
+;; limitations under the License.
+(ns org.apache.storm.starter.clj.bolts
+ (:require [org.apache.storm
+ [clojure :refer :all]
+ [config :refer :all]
+ [log :refer :all]])
+ (:import [org.apache.storm.starter.tools
+NthLastModifiedTimeTracker SlidingWindowCounter
+Rankings RankableObjectWithFields]
+ [org.apache.storm.utils TupleUtils]))
+
+(defbolt rolling-count-bolt ["obj" "count" "actualWindowLengthInSeconds"]
+ {:prepare true
+ :params [window-length emit-frequency]
+ :conf {TOPOLOGY-TICK-TUPLE-FREQ-SECS emit-frequency}}
+ [conf context collector]
+ (let [num-windows (/ window-length emit-frequency)
+counter (SlidingWindowCounter. num-windows)
+tracker (NthLastModifiedTimeTracker. num-windows)]
+(bolt
+ (execute [{word "word" :as tuple}]
+ (if (TupleUtils/isTick tuple)
+ (let [counts (.getCountsThenAdvanceWindow counter)
+ actual-window-length (.secondsSinceOldestModification
tracker)]
+ (log-debug "Received tick tuple, triggering emit of current
window counts")
+ (.markAsModified tracker)
+ (doseq [[obj count] counts]
+ (emit-bolt! collector [obj count actual-window-length])))
+ (do
+ (.incrementCount counter word)
+ (ack! collector tuple)))
+
+(defmacro update-rankings [tuple & body]
+ `(if (TupleUtils/isTick ~tuple)
+ (do
+ (log-debug "Received tick tuple, triggering emit of current
rankings")
+ (emit-bolt! ~'collector [(.copy ~'rankings)])
--- End diff --
My previous comment applies to `collector` and `rankings` as well
> Clojure documentation and examples
> --
>
> Key: STORM-515
> URL: https://issues.apache.org/jira/browse/STORM-515
> Project: Apache Storm
> Issue Type: Improvement
> Components: documentation, examples
>Affects Versions: 0.9.2-incubating
>Reporter: Dmitri Sotnikov
> Labels: newbie
>
> Clojure storm-starter example is extremely basic
> (https://github.com/apache/storm/blob/master/examples/storm-starter/src/clj/storm/starter/clj/word_count.clj)
> and doesn't demonstrate many use cases.
> Clojure documentation page
> (http://storm.incubator.apache.org/documentation/Clojure-DSL.html) has
> unreadable code samples, and glosses over many details in how bolts are
> setup, how to initialize parametarized bolts, where the requires such as
> execute come from, and so on.
> It would be nice to see clear documentation with properly documented examples
> for each of the use cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-515: Clojure documentation and examples
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/1262#discussion_r58153392
--- Diff:
examples/storm-starter/src/clj/org/apache/storm/starter/clj/bolts.clj ---
@@ -0,0 +1,78 @@
+;; Licensed to the Apache Software Foundation (ASF) under one
+;; or more contributor license agreements. See the NOTICE file
+;; distributed with this work for additional information
+;; regarding copyright ownership. The ASF licenses this file
+;; to you under the Apache License, Version 2.0 (the
+;; "License"); you may not use this file except in compliance
+;; with the License. You may obtain a copy of the License at
+;;
+;; http://www.apache.org/licenses/LICENSE-2.0
+;;
+;; Unless required by applicable law or agreed to in writing, software
+;; distributed under the License is distributed on an "AS IS" BASIS,
+;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+;; See the License for the specific language governing permissions and
+;; limitations under the License.
+(ns org.apache.storm.starter.clj.bolts
+ (:require [org.apache.storm
+ [clojure :refer :all]
+ [config :refer :all]
+ [log :refer :all]])
+ (:import [org.apache.storm.starter.tools
+NthLastModifiedTimeTracker SlidingWindowCounter
+Rankings RankableObjectWithFields]
+ [org.apache.storm.utils TupleUtils]))
+
+(defbolt rolling-count-bolt ["obj" "count" "actualWindowLengthInSeconds"]
+ {:prepare true
+ :params [window-length emit-frequency]
+ :conf {TOPOLOGY-TICK-TUPLE-FREQ-SECS emit-frequency}}
+ [conf context collector]
+ (let [num-windows (/ window-length emit-frequency)
+counter (SlidingWindowCounter. num-windows)
+tracker (NthLastModifiedTimeTracker. num-windows)]
+(bolt
+ (execute [{word "word" :as tuple}]
+ (if (TupleUtils/isTick tuple)
+ (let [counts (.getCountsThenAdvanceWindow counter)
+ actual-window-length (.secondsSinceOldestModification
tracker)]
+ (log-debug "Received tick tuple, triggering emit of current
window counts")
+ (.markAsModified tracker)
+ (doseq [[obj count] counts]
+ (emit-bolt! collector [obj count actual-window-length])))
+ (do
+ (.incrementCount counter word)
+ (ack! collector tuple)))
+
+(defmacro update-rankings [tuple & body]
+ `(if (TupleUtils/isTick ~tuple)
+ (do
+ (log-debug "Received tick tuple, triggering emit of current
rankings")
+ (emit-bolt! ~'collector [(.copy ~'rankings)])
--- End diff --
My previous comment applies to `collector` and `rankings` as well
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-1668) Flux silently fails while setting a non-existent property.
[ https://issues.apache.org/jira/browse/STORM-1668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220967#comment-15220967 ] ASF GitHub Bot commented on STORM-1668: --- Github user hustfxj commented on the pull request: https://github.com/apache/storm/pull/1281#issuecomment-204199412 great +1 > Flux silently fails while setting a non-existent property. > -- > > Key: STORM-1668 > URL: https://issues.apache.org/jira/browse/STORM-1668 > Project: Apache Storm > Issue Type: Bug > Components: Flux >Affects Versions: 1.0.0, 2.0.0 >Reporter: Priyank Shah >Assignee: P. Taylor Goetz > Fix For: 1.0.0 > > > Currently, if a yaml file has a property with a name that does not exist on > the java topology component object then flux silently fails by logging a > message and does not throw an exception. This needs to be changed so that > flux throws the exception failing topology submission so that user can take > corrective action. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1668: Fix silent failing of flux for set...
Github user hustfxj commented on the pull request: https://github.com/apache/storm/pull/1281#issuecomment-204199412 great +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-515) Clojure documentation and examples
[ https://issues.apache.org/jira/browse/STORM-515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220964#comment-15220964 ] ASF GitHub Bot commented on STORM-515: -- Github user brymaven commented on the pull request: https://github.com/apache/storm/pull/1262#issuecomment-204199193 Good suggestions. I fixed the macros. > Clojure documentation and examples > -- > > Key: STORM-515 > URL: https://issues.apache.org/jira/browse/STORM-515 > Project: Apache Storm > Issue Type: Improvement > Components: documentation, examples >Affects Versions: 0.9.2-incubating >Reporter: Dmitri Sotnikov > Labels: newbie > > Clojure storm-starter example is extremely basic > (https://github.com/apache/storm/blob/master/examples/storm-starter/src/clj/storm/starter/clj/word_count.clj) > and doesn't demonstrate many use cases. > Clojure documentation page > (http://storm.incubator.apache.org/documentation/Clojure-DSL.html) has > unreadable code samples, and glosses over many details in how bolts are > setup, how to initialize parametarized bolts, where the requires such as > execute come from, and so on. > It would be nice to see clear documentation with properly documented examples > for each of the use cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-515: Clojure documentation and examples
Github user brymaven commented on the pull request: https://github.com/apache/storm/pull/1262#issuecomment-204199193 Good suggestions. I fixed the macros. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1279) port backtype.storm.daemon.supervisor to java
[ https://issues.apache.org/jira/browse/STORM-1279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220961#comment-15220961 ] ASF GitHub Bot commented on STORM-1279: --- Github user hustfxj commented on the pull request: https://github.com/apache/storm/pull/1257#issuecomment-204198660 @revans2 Thank you again. In fact I can't do some testing with profiling options due to my jdk. > port backtype.storm.daemon.supervisor to java > - > > Key: STORM-1279 > URL: https://issues.apache.org/jira/browse/STORM-1279 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: John Fang > Labels: java-migration, jstorm-merger > Attachments: Discussion about supervisor.pdf > > > https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/daemon/supervisor > as an example > backtype.storm.event usage should be replaced with built-in java threadpools. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1279] port backtype.storm.daemon.superv...
Github user hustfxj commented on the pull request: https://github.com/apache/storm/pull/1257#issuecomment-204198660 @revans2 Thank you again. In fact I can't do some testing with profiling options due to my jdk. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1632) Disable event logging by default
[ https://issues.apache.org/jira/browse/STORM-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220960#comment-15220960 ] ASF GitHub Bot commented on STORM-1632: --- Github user roshannaik commented on the pull request: https://github.com/apache/storm/pull/1217#issuecomment-204198533 Update: Have fixes for pts 1 2,3 & 4 raised by Arun. To support the changes in the component level page would have required a change the thrift spec to add the additional info in the response object. However thanks to help for @harshach was able to find an workaround to avoid that. Now the tooltips are shown if both these conditions are met: debug button is disabled and the event loggers are disabled. > Disable event logging by default > > > Key: STORM-1632 > URL: https://issues.apache.org/jira/browse/STORM-1632 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Reporter: Roshan Naik >Assignee: Roshan Naik >Priority: Blocker > Attachments: BasicTopology.java > > > EventLogging has performance penalty. For a simple speed of light topology > with a single instances of a spout and a bolt, disabling event logging > delivers a 7% to 9% perf improvement (with acker count =1) > Event logging can be enabled when there is need to do debug, but turned off > by default. > **Update:** with acker=0 the observed impact was much higher... **25%** > faster when event loggers = 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1632 Disable event logging by default
Github user roshannaik commented on the pull request: https://github.com/apache/storm/pull/1217#issuecomment-204198533 Update: Have fixes for pts 1 2,3 & 4 raised by Arun. To support the changes in the component level page would have required a change the thrift spec to add the additional info in the response object. However thanks to help for @harshach was able to find an workaround to avoid that. Now the tooltips are shown if both these conditions are met: debug button is disabled and the event loggers are disabled. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Code review help for a small pull request
Hi, I have a small pull request for review. Its a one-line functionality change with some small changes for making 3 warnings more helpful. Would appreciate if someone can help me get this in. Here is the PR: https://github.com/apache/storm/pull/1288 Thanks Sachin
[GitHub] storm pull request: KafkaSpout - Improve "_lostMessageCount" metri...
GitHub user sachingsachin opened a pull request: https://github.com/apache/storm/pull/1288 KafkaSpout - Improve "_lostMessageCount" metric for failed tuples Improve "_lostMessageCount" metric if a failed offset is beyond the configured "_spoutConfig.maxOffsetBehind" Other small changes: 1) Rename offset to lowestValidOffset (Avoids confusion with generically-named variable "offset") 2) Move a warning to its proper place (no point warning if there are no omitted offsets) 3) Improve a second warning to convey the magnitude of lost offsets. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sachingsachin/storm master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1288.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1288 commit 53574d44fe070523ac8e83e7df02e925d86efb06 Author: Sachin Goyal Date: 2016-04-01T00:37:40Z Improve "_lostMessageCount" metric if a failed offset is beyond the configured "_spoutConfig.maxOffsetBehind" Other small changes: 1) Rename offset to lowestValidOffset (Avoids confusion with generically-named variable "offset") 2) Move a warning to its proper place (no point warning if there are no omitted offsets) 3) Improve a second warning to convey the magnitude of lost offsets. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (STORM-1667) Log the IO Exception when deleting worker pid dir
[ https://issues.apache.org/jira/browse/STORM-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-1667: Assignee: Zhuo Liu (was: Jungtaek Lim) > Log the IO Exception when deleting worker pid dir > - > > Key: STORM-1667 > URL: https://issues.apache.org/jira/browse/STORM-1667 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.0 >Reporter: Zhuo Liu >Assignee: Zhuo Liu > Fix For: 1.0.0 > > > There is a race condition since shutdown-worker is called by both > sync-processes and synchronize-supervisor in supervisor. One pid directory > might have been already deleted when rmr-as-user tries to delete it. > Therefore, we should not let the FileNotFoundException get thrown and crash > the supervisor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1671) Enable logviewer to delete directory with no yaml file
[ https://issues.apache.org/jira/browse/STORM-1671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220914#comment-15220914 ] ASF GitHub Bot commented on STORM-1671: --- Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/1285#issuecomment-204185356 +1 with assuming manual test has passed. @zhuoliu Would you like to apply this to 1.0.0? If then adding epic to issue would be better to classify. > Enable logviewer to delete directory with no yaml file > --- > > Key: STORM-1671 > URL: https://issues.apache.org/jira/browse/STORM-1671 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Affects Versions: 1.0.0, 2.0.0 >Reporter: Zhuo Liu >Assignee: Zhuo Liu > Fix For: 1.0.0, 2.0.0 > > > For those old and dead worker directories, in some weird case, there is no > yaml file in it. We should enable logviewer to delete them (any dir that has/ > has no yaml file in it).. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1671] Enable logviewer to delete a dir ...
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/1285#issuecomment-204185356 +1 with assuming manual test has passed. @zhuoliu Would you like to apply this to 1.0.0? If then adding epic to issue would be better to classify. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (STORM-1667) Log the IO Exception when deleting worker pid dir
[ https://issues.apache.org/jira/browse/STORM-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-1667: Add Epic cause it's a bug and target version is 1.0.0. > Log the IO Exception when deleting worker pid dir > - > > Key: STORM-1667 > URL: https://issues.apache.org/jira/browse/STORM-1667 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.0 >Reporter: Zhuo Liu >Assignee: Zhuo Liu > Fix For: 1.0.0 > > > There is a race condition since shutdown-worker is called by both > sync-processes and synchronize-supervisor in supervisor. One pid directory > might have been already deleted when rmr-as-user tries to delete it. > Therefore, we should not let the FileNotFoundException get thrown and crash > the supervisor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1667) Log the IO Exception when deleting worker pid dir
[ https://issues.apache.org/jira/browse/STORM-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220891#comment-15220891 ] ASF GitHub Bot commented on STORM-1667: --- Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/1280#issuecomment-204181100 +1 Great finding. > Log the IO Exception when deleting worker pid dir > - > > Key: STORM-1667 > URL: https://issues.apache.org/jira/browse/STORM-1667 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.0 >Reporter: Zhuo Liu >Assignee: Zhuo Liu > Fix For: 1.0.0 > > > There is a race condition since shutdown-worker is called by both > sync-processes and synchronize-supervisor in supervisor. One pid directory > might have been already deleted when rmr-as-user tries to delete it. > Therefore, we should not let the FileNotFoundException get thrown and crash > the supervisor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1667] Log the IO exception when deletin...
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/1280#issuecomment-204181100 +1 Great finding. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-1668: Fix silent failing of flux for set...
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/1281#issuecomment-204179234 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1668) Flux silently fails while setting a non-existent property.
[ https://issues.apache.org/jira/browse/STORM-1668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220883#comment-15220883 ] ASF GitHub Bot commented on STORM-1668: --- Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/1281#issuecomment-204179234 +1 > Flux silently fails while setting a non-existent property. > -- > > Key: STORM-1668 > URL: https://issues.apache.org/jira/browse/STORM-1668 > Project: Apache Storm > Issue Type: Bug > Components: Flux >Affects Versions: 1.0.0, 2.0.0 >Reporter: Priyank Shah >Assignee: P. Taylor Goetz > Fix For: 1.0.0 > > > Currently, if a yaml file has a property with a name that does not exist on > the java topology component object then flux silently fails by logging a > message and does not throw an exception. This needs to be changed so that > flux throws the exception failing topology submission so that user can take > corrective action. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1668) Flux silently fails while setting a non-existent property.
[ https://issues.apache.org/jira/browse/STORM-1668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220882#comment-15220882 ] ASF GitHub Bot commented on STORM-1668: --- Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/1282#issuecomment-204179169 +1 > Flux silently fails while setting a non-existent property. > -- > > Key: STORM-1668 > URL: https://issues.apache.org/jira/browse/STORM-1668 > Project: Apache Storm > Issue Type: Bug > Components: Flux >Affects Versions: 1.0.0, 2.0.0 >Reporter: Priyank Shah >Assignee: P. Taylor Goetz > Fix For: 1.0.0 > > > Currently, if a yaml file has a property with a name that does not exist on > the java topology component object then flux silently fails by logging a > message and does not throw an exception. This needs to be changed so that > flux throws the exception failing topology submission so that user can take > corrective action. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1668: Fix silent failing of flux for set...
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/1282#issuecomment-204179169 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Addresses network flood from KafkaSpout to kaf...
GitHub user moesol opened a pull request: https://github.com/apache/storm/pull/1287 Addresses network flood from KafkaSpout to kafka server. * Allows minBytes in fetch request to be configured from KafkaConfig.fetchMinBytes. * Defaults new configuration KafkaConfig.fetchMinBytes to 1. * Defaults fetchMaxWait to 100ms instead of 1ms. Discovered 30 megabits of traffic flowing between a set of KafkaSpouts and our kafka servers even though no Kafka messages were moving. Using the wireshark kafka dissector, we were able to see that each FetchRequest had maxWait set to 1 and minBytes set to 0. When binBytes is set to 0 the kafka server responds immediately when there are no messages. In turn the KafkaSpout polls without any delay causing a constant stream of FetchRequest/ FetchResponse messages. Using a non-KafkaSpout client had a similar traffic pattern with two key differences 1) minBytes was 1 2) maxWait was 100 With these FetchRequest parameters and no messages flowing, the kafka server delays the FetchResponse by 100 ms. This reduces the network traffic from megabits to the low kilobits. It also reduced the CPU utilization of our kafka server from 140% to 2%. Hopefully the risk of this change is low because the old behavior can be restored using the API by setting KafkaConfig.fetchMaxWait to 1 KafkaConfig.fetchMinBytes to 0 You can merge this pull request into a Git repository by running: $ git pull https://github.com/MoebiusSolutions/storm 0.9.3-branch Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1287.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1287 commit 66c38ad28ce597a4df6c05db3264ff94960d6764 Author: Robert Hastings Date: 2016-03-31T23:14:47Z Addresses network flood from KafkaSpout to kafka server. * Allows minBytes in fetch request to be configured from KafkaConfig.fetchMinBytes. * Defaults new configuration KafkaConfig.fetchMinBytes to 1. * Defaults fetchMaxWait to 100ms instead of 1ms. Discovered 30 megabits of traffic flowing between a set of KafkaSpouts and our kafka servers even though no Kafka messages were moving. Using the wireshark kafka dissector, we were able to see that each FetchRequest had maxWait set to 1 and minBytes set to 0. When binBytes is set to 0 the kafka server responds immediately when there are no messages. In turn the KafkaSpout polls without any delay causing a constant stream of FetchRequest/ FetchResponse messages. Using a non-KafkaSpout client had a similar traffic pattern with two key differences 1) minBytes was 1 2) maxWait was 100 With these FetchRequest parameters and no messages flowing, the kafka server delays the FetchResponse by 100 ms. This reduces the network traffic from megabits to the low kilobits. It also reduced the CPU utilization of our kafka server from 140% to 2%. Hopefully the risk of this change is low because the old behavior can be restored using the API by setting KafkaConfig.fetchMaxWait to 1 KafkaConfig.fetchMinBytes to 0 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1279) port backtype.storm.daemon.supervisor to java
[ https://issues.apache.org/jira/browse/STORM-1279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220842#comment-15220842 ] ASF GitHub Bot commented on STORM-1279: --- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/1257#issuecomment-204167455 Found a few more bugs with manual testing. I would like to do some more with other profiling options that are not on by default but this should fix the default jstack/heap dump profiling. > port backtype.storm.daemon.supervisor to java > - > > Key: STORM-1279 > URL: https://issues.apache.org/jira/browse/STORM-1279 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: John Fang > Labels: java-migration, jstorm-merger > Attachments: Discussion about supervisor.pdf > > > https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/daemon/supervisor > as an example > backtype.storm.event usage should be replaced with built-in java threadpools. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1279] port backtype.storm.daemon.superv...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/1257#issuecomment-204167455 Found a few more bugs with manual testing. I would like to do some more with other profiling options that are not on by default but this should fix the default jstack/heap dump profiling. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-1279] port backtype.storm.daemon.superv...
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1257#discussion_r58142341
--- Diff:
storm-core/src/jvm/org/apache/storm/daemon/supervisor/timer/RunProfilerActions.java
---
@@ -0,0 +1,214 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.daemon.supervisor.timer;
+
+import com.google.common.collect.Lists;
+import org.apache.storm.Config;
+import org.apache.storm.cluster.IStormClusterState;
+import org.apache.storm.daemon.supervisor.SupervisorData;
+import org.apache.storm.daemon.supervisor.SupervisorUtils;
+import org.apache.storm.generated.ProfileAction;
+import org.apache.storm.generated.ProfileRequest;
+import org.apache.storm.utils.ConfigUtils;
+import org.apache.storm.utils.Utils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.*;
+
+public class RunProfilerActions implements Runnable {
+private static Logger LOG =
LoggerFactory.getLogger(RunProfilerActions.class);
+
+private Map conf;
+private IStormClusterState stormClusterState;
+private String hostName;
+
+private String profileCmd;
+
+private SupervisorData supervisorData;
+
+private class ActionExitCallback implements Utils.ExitCodeCallable {
+private String stormId;
+private ProfileRequest profileRequest;
+private String logPrefix;
+private boolean stop;
+
+public ActionExitCallback(String stormId, ProfileRequest
profileRequest, String logPrefix, boolean stop) {
+this.stormId = stormId;
+this.profileRequest = profileRequest;
+this.logPrefix = logPrefix;
+this.stop = stop;
+}
+
+@Override
+public Object call() throws Exception {
+return null;
+}
+
+@Override
+public Object call(int exitCode) {
+LOG.info("{} profile-action exited for {}", logPrefix,
exitCode);
+try {
+if (stop)
+
stormClusterState.deleteTopologyProfileRequests(stormId, profileRequest);
+} catch (Exception e) {
+LOG.warn("failed delete profileRequest: " +
profileRequest);
+}
+return null;
+}
+}
+
+public RunProfilerActions(SupervisorData supervisorData) {
+this.conf = supervisorData.getConf();
+this.stormClusterState = supervisorData.getStormClusterState();
+this.hostName = supervisorData.getHostName();
+this.profileCmd = (String)
(conf.get(Config.WORKER_PROFILER_COMMAND));
+this.supervisorData = supervisorData;
+}
+
+@Override
+public void run() {
+Map> stormIdToActions =
supervisorData.getStormIdToProfilerActions().get();
+try {
+for (Map.Entry> entry :
stormIdToActions.entrySet()) {
+String stormId = entry.getKey();
+List requests = entry.getValue();
+if (requests != null) {
+for (ProfileRequest profileRequest : requests) {
+if
(profileRequest.get_nodeInfo().get_node().equals(hostName)) {
+boolean stop = System.currentTimeMillis() >
profileRequest.get_time_stamp();
+Long port =
profileRequest.get_nodeInfo().get_port().iterator().next();
+String targetDir =
ConfigUtils.workerArtifactsRoot(conf, String.valueOf(port));
+Map stormConf =
ConfigUtils.readSupervisorStormConf(conf, stormId);
+
+String user = null;
+if
(stormConf.get(Config.
[jira] [Commented] (STORM-1279) port backtype.storm.daemon.supervisor to java
[
https://issues.apache.org/jira/browse/STORM-1279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220831#comment-15220831
]
ASF GitHub Bot commented on STORM-1279:
---
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1257#discussion_r58142341
--- Diff:
storm-core/src/jvm/org/apache/storm/daemon/supervisor/timer/RunProfilerActions.java
---
@@ -0,0 +1,214 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.daemon.supervisor.timer;
+
+import com.google.common.collect.Lists;
+import org.apache.storm.Config;
+import org.apache.storm.cluster.IStormClusterState;
+import org.apache.storm.daemon.supervisor.SupervisorData;
+import org.apache.storm.daemon.supervisor.SupervisorUtils;
+import org.apache.storm.generated.ProfileAction;
+import org.apache.storm.generated.ProfileRequest;
+import org.apache.storm.utils.ConfigUtils;
+import org.apache.storm.utils.Utils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.*;
+
+public class RunProfilerActions implements Runnable {
+private static Logger LOG =
LoggerFactory.getLogger(RunProfilerActions.class);
+
+private Map conf;
+private IStormClusterState stormClusterState;
+private String hostName;
+
+private String profileCmd;
+
+private SupervisorData supervisorData;
+
+private class ActionExitCallback implements Utils.ExitCodeCallable {
+private String stormId;
+private ProfileRequest profileRequest;
+private String logPrefix;
+private boolean stop;
+
+public ActionExitCallback(String stormId, ProfileRequest
profileRequest, String logPrefix, boolean stop) {
+this.stormId = stormId;
+this.profileRequest = profileRequest;
+this.logPrefix = logPrefix;
+this.stop = stop;
+}
+
+@Override
+public Object call() throws Exception {
+return null;
+}
+
+@Override
+public Object call(int exitCode) {
+LOG.info("{} profile-action exited for {}", logPrefix,
exitCode);
+try {
+if (stop)
+
stormClusterState.deleteTopologyProfileRequests(stormId, profileRequest);
+} catch (Exception e) {
+LOG.warn("failed delete profileRequest: " +
profileRequest);
+}
+return null;
+}
+}
+
+public RunProfilerActions(SupervisorData supervisorData) {
+this.conf = supervisorData.getConf();
+this.stormClusterState = supervisorData.getStormClusterState();
+this.hostName = supervisorData.getHostName();
+this.profileCmd = (String)
(conf.get(Config.WORKER_PROFILER_COMMAND));
+this.supervisorData = supervisorData;
+}
+
+@Override
+public void run() {
+Map> stormIdToActions =
supervisorData.getStormIdToProfilerActions().get();
+try {
+for (Map.Entry> entry :
stormIdToActions.entrySet()) {
+String stormId = entry.getKey();
+List requests = entry.getValue();
+if (requests != null) {
+for (ProfileRequest profileRequest : requests) {
+if
(profileRequest.get_nodeInfo().get_node().equals(hostName)) {
+boolean stop = System.currentTimeMillis() >
profileRequest.get_time_stamp();
+Long port =
profileRequest.get_nodeInfo().get_port().iterator().next();
+String targetDir =
ConfigUtils.workerArtifac
[jira] [Commented] (STORM-1279) port backtype.storm.daemon.supervisor to java
[
https://issues.apache.org/jira/browse/STORM-1279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220803#comment-15220803
]
ASF GitHub Bot commented on STORM-1279:
---
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1257#discussion_r58140035
--- Diff:
storm-core/src/jvm/org/apache/storm/daemon/supervisor/timer/RunProfilerActions.java
---
@@ -0,0 +1,214 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.daemon.supervisor.timer;
+
+import com.google.common.collect.Lists;
+import org.apache.storm.Config;
+import org.apache.storm.cluster.IStormClusterState;
+import org.apache.storm.daemon.supervisor.SupervisorData;
+import org.apache.storm.daemon.supervisor.SupervisorUtils;
+import org.apache.storm.generated.ProfileAction;
+import org.apache.storm.generated.ProfileRequest;
+import org.apache.storm.utils.ConfigUtils;
+import org.apache.storm.utils.Utils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.*;
+
+public class RunProfilerActions implements Runnable {
+private static Logger LOG =
LoggerFactory.getLogger(RunProfilerActions.class);
+
+private Map conf;
+private IStormClusterState stormClusterState;
+private String hostName;
+
+private String profileCmd;
+
+private SupervisorData supervisorData;
+
+private class ActionExitCallback implements Utils.ExitCodeCallable {
+private String stormId;
+private ProfileRequest profileRequest;
+private String logPrefix;
+private boolean stop;
+
+public ActionExitCallback(String stormId, ProfileRequest
profileRequest, String logPrefix, boolean stop) {
+this.stormId = stormId;
+this.profileRequest = profileRequest;
+this.logPrefix = logPrefix;
+this.stop = stop;
+}
+
+@Override
+public Object call() throws Exception {
+return null;
+}
+
+@Override
+public Object call(int exitCode) {
+LOG.info("{} profile-action exited for {}", logPrefix,
exitCode);
+try {
+if (stop)
+
stormClusterState.deleteTopologyProfileRequests(stormId, profileRequest);
+} catch (Exception e) {
+LOG.warn("failed delete profileRequest: " +
profileRequest);
+}
+return null;
+}
+}
+
+public RunProfilerActions(SupervisorData supervisorData) {
+this.conf = supervisorData.getConf();
+this.stormClusterState = supervisorData.getStormClusterState();
+this.hostName = supervisorData.getHostName();
+this.profileCmd = (String)
(conf.get(Config.WORKER_PROFILER_COMMAND));
+this.supervisorData = supervisorData;
+}
+
+@Override
+public void run() {
+Map> stormIdToActions =
supervisorData.getStormIdToProfilerActions().get();
+try {
+for (Map.Entry> entry :
stormIdToActions.entrySet()) {
+String stormId = entry.getKey();
+List requests = entry.getValue();
+if (requests != null) {
+for (ProfileRequest profileRequest : requests) {
+if
(profileRequest.get_nodeInfo().get_node().equals(hostName)) {
+boolean stop = System.currentTimeMillis() >
profileRequest.get_time_stamp();
+Long port =
profileRequest.get_nodeInfo().get_port().iterator().next();
+String targetDir =
ConfigUtils.workerArtifac
[GitHub] storm pull request: [STORM-1279] port backtype.storm.daemon.superv...
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1257#discussion_r58140035
--- Diff:
storm-core/src/jvm/org/apache/storm/daemon/supervisor/timer/RunProfilerActions.java
---
@@ -0,0 +1,214 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.daemon.supervisor.timer;
+
+import com.google.common.collect.Lists;
+import org.apache.storm.Config;
+import org.apache.storm.cluster.IStormClusterState;
+import org.apache.storm.daemon.supervisor.SupervisorData;
+import org.apache.storm.daemon.supervisor.SupervisorUtils;
+import org.apache.storm.generated.ProfileAction;
+import org.apache.storm.generated.ProfileRequest;
+import org.apache.storm.utils.ConfigUtils;
+import org.apache.storm.utils.Utils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.*;
+
+public class RunProfilerActions implements Runnable {
+private static Logger LOG =
LoggerFactory.getLogger(RunProfilerActions.class);
+
+private Map conf;
+private IStormClusterState stormClusterState;
+private String hostName;
+
+private String profileCmd;
+
+private SupervisorData supervisorData;
+
+private class ActionExitCallback implements Utils.ExitCodeCallable {
+private String stormId;
+private ProfileRequest profileRequest;
+private String logPrefix;
+private boolean stop;
+
+public ActionExitCallback(String stormId, ProfileRequest
profileRequest, String logPrefix, boolean stop) {
+this.stormId = stormId;
+this.profileRequest = profileRequest;
+this.logPrefix = logPrefix;
+this.stop = stop;
+}
+
+@Override
+public Object call() throws Exception {
+return null;
+}
+
+@Override
+public Object call(int exitCode) {
+LOG.info("{} profile-action exited for {}", logPrefix,
exitCode);
+try {
+if (stop)
+
stormClusterState.deleteTopologyProfileRequests(stormId, profileRequest);
+} catch (Exception e) {
+LOG.warn("failed delete profileRequest: " +
profileRequest);
+}
+return null;
+}
+}
+
+public RunProfilerActions(SupervisorData supervisorData) {
+this.conf = supervisorData.getConf();
+this.stormClusterState = supervisorData.getStormClusterState();
+this.hostName = supervisorData.getHostName();
+this.profileCmd = (String)
(conf.get(Config.WORKER_PROFILER_COMMAND));
+this.supervisorData = supervisorData;
+}
+
+@Override
+public void run() {
+Map> stormIdToActions =
supervisorData.getStormIdToProfilerActions().get();
+try {
+for (Map.Entry> entry :
stormIdToActions.entrySet()) {
+String stormId = entry.getKey();
+List requests = entry.getValue();
+if (requests != null) {
+for (ProfileRequest profileRequest : requests) {
+if
(profileRequest.get_nodeInfo().get_node().equals(hostName)) {
+boolean stop = System.currentTimeMillis() >
profileRequest.get_time_stamp();
+Long port =
profileRequest.get_nodeInfo().get_port().iterator().next();
+String targetDir =
ConfigUtils.workerArtifactsRoot(conf, String.valueOf(port));
--- End diff --
This has to have the topology ID as a part of it.
```
String targetDir =
ConfigUtils.workerArtifactsRoot(conf, stormId, port);
```
---
If your
[jira] [Commented] (STORM-1279) port backtype.storm.daemon.supervisor to java
[
https://issues.apache.org/jira/browse/STORM-1279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220769#comment-15220769
]
ASF GitHub Bot commented on STORM-1279:
---
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1257#discussion_r58137861
--- Diff:
storm-core/src/jvm/org/apache/storm/daemon/supervisor/timer/RunProfilerActions.java
---
@@ -0,0 +1,214 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.daemon.supervisor.timer;
+
+import com.google.common.collect.Lists;
+import org.apache.storm.Config;
+import org.apache.storm.cluster.IStormClusterState;
+import org.apache.storm.daemon.supervisor.SupervisorData;
+import org.apache.storm.daemon.supervisor.SupervisorUtils;
+import org.apache.storm.generated.ProfileAction;
+import org.apache.storm.generated.ProfileRequest;
+import org.apache.storm.utils.ConfigUtils;
+import org.apache.storm.utils.Utils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.*;
+
+public class RunProfilerActions implements Runnable {
+private static Logger LOG =
LoggerFactory.getLogger(RunProfilerActions.class);
+
+private Map conf;
+private IStormClusterState stormClusterState;
+private String hostName;
+
+private String profileCmd;
+
+private SupervisorData supervisorData;
+
+private class ActionExitCallback implements Utils.ExitCodeCallable {
+private String stormId;
+private ProfileRequest profileRequest;
+private String logPrefix;
+private boolean stop;
+
+public ActionExitCallback(String stormId, ProfileRequest
profileRequest, String logPrefix, boolean stop) {
+this.stormId = stormId;
+this.profileRequest = profileRequest;
+this.logPrefix = logPrefix;
+this.stop = stop;
+}
+
+@Override
+public Object call() throws Exception {
+return null;
+}
+
+@Override
+public Object call(int exitCode) {
+LOG.info("{} profile-action exited for {}", logPrefix,
exitCode);
+try {
+if (stop)
+
stormClusterState.deleteTopologyProfileRequests(stormId, profileRequest);
+} catch (Exception e) {
+LOG.warn("failed delete profileRequest: " +
profileRequest);
+}
+return null;
+}
+}
+
+public RunProfilerActions(SupervisorData supervisorData) {
+this.conf = supervisorData.getConf();
+this.stormClusterState = supervisorData.getStormClusterState();
+this.hostName = supervisorData.getHostName();
+this.profileCmd = (String)
(conf.get(Config.WORKER_PROFILER_COMMAND));
--- End diff --
The original code had this relative to the `storm.home/bin`
```
profile-cmd (str (clojure.java.io/file storm-home
"bin"
(conf
WORKER-PROFILER-COMMAND)))
```
Probably should be
```
String stormHome = System.getProperty("storm.home");
this.profileCmd = stormHome + Utils.FILE_PATH_SEPARATOR + "bin" +
Utils.FILE_PATH_SEPARATOR + conf.get(Config.WORKER_PROFILER_COMMAND));
```
> port backtype.storm.daemon.supervisor to java
> -
>
> Key: STORM-1279
> URL: https://issues.apache.org/jira/browse/STORM-1279
> Project: Apache Storm
> Issue Type: New Feature
> Components: storm-core
>Reporter: Robe
[GitHub] storm pull request: [STORM-1279] port backtype.storm.daemon.superv...
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1257#discussion_r58137861
--- Diff:
storm-core/src/jvm/org/apache/storm/daemon/supervisor/timer/RunProfilerActions.java
---
@@ -0,0 +1,214 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.daemon.supervisor.timer;
+
+import com.google.common.collect.Lists;
+import org.apache.storm.Config;
+import org.apache.storm.cluster.IStormClusterState;
+import org.apache.storm.daemon.supervisor.SupervisorData;
+import org.apache.storm.daemon.supervisor.SupervisorUtils;
+import org.apache.storm.generated.ProfileAction;
+import org.apache.storm.generated.ProfileRequest;
+import org.apache.storm.utils.ConfigUtils;
+import org.apache.storm.utils.Utils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.*;
+
+public class RunProfilerActions implements Runnable {
+private static Logger LOG =
LoggerFactory.getLogger(RunProfilerActions.class);
+
+private Map conf;
+private IStormClusterState stormClusterState;
+private String hostName;
+
+private String profileCmd;
+
+private SupervisorData supervisorData;
+
+private class ActionExitCallback implements Utils.ExitCodeCallable {
+private String stormId;
+private ProfileRequest profileRequest;
+private String logPrefix;
+private boolean stop;
+
+public ActionExitCallback(String stormId, ProfileRequest
profileRequest, String logPrefix, boolean stop) {
+this.stormId = stormId;
+this.profileRequest = profileRequest;
+this.logPrefix = logPrefix;
+this.stop = stop;
+}
+
+@Override
+public Object call() throws Exception {
+return null;
+}
+
+@Override
+public Object call(int exitCode) {
+LOG.info("{} profile-action exited for {}", logPrefix,
exitCode);
+try {
+if (stop)
+
stormClusterState.deleteTopologyProfileRequests(stormId, profileRequest);
+} catch (Exception e) {
+LOG.warn("failed delete profileRequest: " +
profileRequest);
+}
+return null;
+}
+}
+
+public RunProfilerActions(SupervisorData supervisorData) {
+this.conf = supervisorData.getConf();
+this.stormClusterState = supervisorData.getStormClusterState();
+this.hostName = supervisorData.getHostName();
+this.profileCmd = (String)
(conf.get(Config.WORKER_PROFILER_COMMAND));
--- End diff --
The original code had this relative to the `storm.home/bin`
```
profile-cmd (str (clojure.java.io/file storm-home
"bin"
(conf
WORKER-PROFILER-COMMAND)))
```
Probably should be
```
String stormHome = System.getProperty("storm.home");
this.profileCmd = stormHome + Utils.FILE_PATH_SEPARATOR + "bin" +
Utils.FILE_PATH_SEPARATOR + conf.get(Config.WORKER_PROFILER_COMMAND));
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-515) Clojure documentation and examples
[ https://issues.apache.org/jira/browse/STORM-515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220733#comment-15220733 ] ASF GitHub Bot commented on STORM-515: -- Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/1262#issuecomment-204148932 Looks good for the most part. > Clojure documentation and examples > -- > > Key: STORM-515 > URL: https://issues.apache.org/jira/browse/STORM-515 > Project: Apache Storm > Issue Type: Improvement > Components: documentation, examples >Affects Versions: 0.9.2-incubating >Reporter: Dmitri Sotnikov > Labels: newbie > > Clojure storm-starter example is extremely basic > (https://github.com/apache/storm/blob/master/examples/storm-starter/src/clj/storm/starter/clj/word_count.clj) > and doesn't demonstrate many use cases. > Clojure documentation page > (http://storm.incubator.apache.org/documentation/Clojure-DSL.html) has > unreadable code samples, and glosses over many details in how bolts are > setup, how to initialize parametarized bolts, where the requires such as > execute come from, and so on. > It would be nice to see clear documentation with properly documented examples > for each of the use cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-515: Clojure documentation and examples
Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/1262#issuecomment-204148932 Looks good for the most part. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
