[GitHub] storm pull request: [STORM-949] On the topology summary UI page, a...
Github user jerrypeng commented on a diff in the pull request: https://github.com/apache/storm/pull/675#discussion_r36867859 --- Diff: storm-core/src/ui/public/templates/topology-page-template.html --- @@ -313,6 +320,10 @@ /th th class=headerLast error /th + th class=header --- End diff -- +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-949] On the topology summary UI page, a...
Github user jerrypeng commented on the pull request: https://github.com/apache/storm/pull/675#issuecomment-130327785  --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-949) On the topology summary UI page, last shown error should have the time and date
[ https://issues.apache.org/jira/browse/STORM-949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14693612#comment-14693612 ] ASF GitHub Bot commented on STORM-949: -- Github user jerrypeng commented on the pull request: https://github.com/apache/storm/pull/675#issuecomment-130327785  On the topology summary UI page, last shown error should have the time and date --- Key: STORM-949 URL: https://issues.apache.org/jira/browse/STORM-949 Project: Apache Storm Issue Type: Improvement Reporter: Reza Farivar Assignee: Boyang Jerry Peng Priority: Minor Without showing the time and date, it is not clear whether the error is a recent one or an old stale one. (Error text is colored red when it is in last 5 minutes, but this is not obvious and not enough information) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-949) On the topology summary UI page, last shown error should have the time and date
[ https://issues.apache.org/jira/browse/STORM-949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14693609#comment-14693609 ] ASF GitHub Bot commented on STORM-949: -- Github user jerrypeng commented on a diff in the pull request: https://github.com/apache/storm/pull/675#discussion_r36867859 --- Diff: storm-core/src/ui/public/templates/topology-page-template.html --- @@ -313,6 +320,10 @@ /th th class=headerLast error /th + th class=header --- End diff -- +1 On the topology summary UI page, last shown error should have the time and date --- Key: STORM-949 URL: https://issues.apache.org/jira/browse/STORM-949 Project: Apache Storm Issue Type: Improvement Reporter: Reza Farivar Assignee: Boyang Jerry Peng Priority: Minor Without showing the time and date, it is not clear whether the error is a recent one or an old stale one. (Error text is colored red when it is in last 5 minutes, but this is not obvious and not enough information) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-949] On the topology summary UI page, a...
Github user jerrypeng commented on the pull request: https://github.com/apache/storm/pull/675#issuecomment-130347080 fixed the formating issues --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-949) On the topology summary UI page, last shown error should have the time and date
[ https://issues.apache.org/jira/browse/STORM-949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14693695#comment-14693695 ] ASF GitHub Bot commented on STORM-949: -- Github user jerrypeng commented on the pull request: https://github.com/apache/storm/pull/675#issuecomment-130347080 fixed the formating issues On the topology summary UI page, last shown error should have the time and date --- Key: STORM-949 URL: https://issues.apache.org/jira/browse/STORM-949 Project: Apache Storm Issue Type: Improvement Reporter: Reza Farivar Assignee: Boyang Jerry Peng Priority: Minor Without showing the time and date, it is not clear whether the error is a recent one or an old stale one. (Error text is colored red when it is in last 5 minutes, but this is not obvious and not enough information) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-950. Apache Storm website redesign.
Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/670#issuecomment-130444965 I have a few things that I think need to be fixed before we publish this. But rather than enumerate every last little thing and ask you to fix them, I'd rather jump in and fix them myself and let others do the same. I propose we move this off to an empty branch that just contains the storm website code -- similar to the way github pages work. Then anyone could help fix things by opening a pull request against that branch. Then once we are all satisfied enough to publish it, we do so and remove the website code from the main source branches. I don't have a reference handy, but I think ASF INFRA has a way to publish web content from a specially-named branch (similar to how gh-pages branches work). That would eliminate the extra step of generating the site to SVN and committing that way. I'll look into this option some more. How does that sound? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-950. Apache Storm website redesign.
Github user harshach commented on the pull request: https://github.com/apache/storm/pull/670#issuecomment-130447771 @ptgoetz if these issues are layout please list them out we can send a new patch. I don't think website layout needs to be blocked on docs or other fixes. I want to see this published soon rather than make it into feature branch and taking time to fix this. If its about docs we can file follow-up jiras to fix those. Even if its layout issues these should be minor and can be make it into follow-up jiras. It doesn't need to get into feature branch and IMHO its ready get published. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-826] Update KafkaBolt to use the new ka...
Github user zhuoliu commented on the pull request: https://github.com/apache/storm/pull/572#issuecomment-130454722 Thanks @lazyval for the comments, we may file a separate jira to do some follow-up work for this as needed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-594) Auto-Scaling Resources in a Topology
[ https://issues.apache.org/jira/browse/STORM-594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694064#comment-14694064 ] Jon Weygandt commented on STORM-594: Do you have a link to this paper? Auto-Scaling Resources in a Topology Key: STORM-594 URL: https://issues.apache.org/jira/browse/STORM-594 Project: Apache Storm Issue Type: New Feature Reporter: HARSHA BALASUBRAMANIAN Assignee: Pooyan Jamshidi Priority: Minor Attachments: Algorithm for Auto-Scaling.pdf, Project Plan and Scope.pdf Original Estimate: 504h Remaining Estimate: 504h A useful feature missing in Storm topologies is the ability to auto-scale resources, based on a pre-configured metric. The feature proposed here aims to build such a auto-scaling mechanism using a feedback system. A brief overview of the feature is provided here. The finer details of the required components and the scaling algorithm (uses a Feedback System) are provided in the PDFs attached. Brief Overview: Topologies may get created with or (ideally) without parallelism hints and tasks in their bolts and spouts, before submitting them, If auto-scaling is set in the topology (using a Boolean flag), the topology will also get submitted to the auto-scale module. The auto-scale module will read a pre-configured metric (threshold/min) from a configuration file. Using this value, the topology's resources will be modified till the threshold is reached. At each stage in the auto-scale module's execution, feedback from the previous execution will be used to tune the resources. The systems that need to be in place to achieve this are: 1. Metrics which provide the current threshold (no: of acks per minute) for a topology's spouts and bolts. 2. Access to Storm's CLI tool which can change a topology's resources are runtime. 3. A new java or clojure module which runs within the Nimbus daemon or in parallel to it. This will be the auto-scale module. Limitations: (This is not an exhaustive list. More will be added as the design matures. Also, some of the points here may get resolved) To test the feature there will be a number of limitations in the first release. As the feature matures, it will be allowed to scale more 1. The auto-scale module will be limited to a few topologies (maybe 4 or 5 at maximum) 2. New bolts will not be added to scale a topology. This feature will be limited to increasing the resources within the existing topology. 3. Topology resources will not be decreased when it is running at more than the required number (except for a few cases) 4. This feature will work only for long-running topologies where the input threshold can become equal to or greater than the required threshold -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: Storm-166: Nimbus HA design doc and implementa...
Github user Parth-Brahmbhatt commented on the pull request: https://github.com/apache/storm/pull/354#issuecomment-130450409 @revans2 @ptgoetz @harshach I have merged with master one more time. We are still using curator's LeaderLatch recipe. Nimbus discovery is done via an API so all clients don't have to connect to zookeeper. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-950. Apache Storm website redesign.
Github user harshach commented on the pull request: https://github.com/apache/storm/pull/670#issuecomment-130488992 @ptgoetz will look into the fb goog calls and also generating the news feed. As far as separate branch goes thats fine by me but another issue could be that the doc contributions will not be counted towards master branch rather into another branch and also merging those changes into another branch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-980) Re-include storm-kafka tests from Travis CI build
[ https://issues.apache.org/jira/browse/STORM-980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14693924#comment-14693924 ] ASF GitHub Bot commented on STORM-980: -- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/674 Re-include storm-kafka tests from Travis CI build - Key: STORM-980 URL: https://issues.apache.org/jira/browse/STORM-980 Project: Apache Storm Issue Type: Sub-task Components: storm-kafka Reporter: Jungtaek Lim Assignee: Jungtaek Lim Recently [~zhuoliu] reported that randome unit tests failures on storm-kafka may be resolved with STORM-826. I rebuilt test-kafka-test branch 5 times in a row (tested with jdk7, jdk8, so storm-kafka unit tests were run 10 times), and I can see there's no longer storm-kafka's random failure. It is broken at 6th trial, but I've met random failure of storm-core first, so we're leaving decision whether we ignore low probabilities of random failures or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-950. Apache Storm website redesign.
Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/670#issuecomment-130452561 @harshach I'm not suggesting we create a feature branch, I'm suggesting we change the way we publish the site to make it easier for people to contribute to. I forgot to mention it before, but I'm happy to set that up. Some of the issues I see are minor layout issues (twitter feed should be below news, for example), some are maintenance-related (the recent news on the home page should be generated, not hard-coded in HTML), and some are related to ASF policy (we are pulling in content from facebook.com and google.com domains that potentially includes end-user tracking, and I'm not sure if that's allowed -- I'd have to check). I'm not worried about docs/textual content, that can be fixed anytime. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-977) Incorrect signal (-9) when as-user is true
[ https://issues.apache.org/jira/browse/STORM-977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694269#comment-14694269 ] Gabor Liptak commented on STORM-977: https://en.wikipedia.org/wiki/Unix_signal Incorrect signal (-9) when as-user is true -- Key: STORM-977 URL: https://issues.apache.org/jira/browse/STORM-977 Project: Apache Storm Issue Type: Bug Affects Versions: 0.10.0 Reporter: Derek Dagit Priority: Trivial Labels: Newbie https://github.com/apache/storm/blob/544e55cb8ab8878c4af500aab49bd35d4b69cd3e/storm-core/src/clj/backtype/storm/daemon/supervisor.clj#L265 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: Update timer.clj
Github user caofangkun commented on a diff in the pull request: https://github.com/apache/storm/pull/680#discussion_r36937217 --- Diff: storm-core/src/clj/backtype/storm/timer.clj --- @@ -53,8 +53,11 @@ ;; event generation. If any recurring events ;; are scheduled then we will always go ;; through this branch, sleeping only the - ;; exact necessary amount of time. - (Time/sleep (- time-millis (current-time-millis))) + ;; exact necessary amount of time. We give + ;; an upper bound, e.g. 1000 millis, to the + ;; sleeping time, to limit the response time + ;; for detecting any new event within 1 secs. + (Time/sleep (max 1000 (- time-millis (current-time-millis --- End diff -- I think this shoud use ``` min``` to limit the time. ```(Time/sleep (min 1000 (- time-millis (current-time-millis``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-974) [storm-elasticsearch] Introduces Tuple - ES document mapper to get rid of constant field mapping (source, index, type)
[ https://issues.apache.org/jira/browse/STORM-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694633#comment-14694633 ] ASF GitHub Bot commented on STORM-974: -- Github user caofangkun commented on a diff in the pull request: https://github.com/apache/storm/pull/679#discussion_r36938233 --- Diff: external/storm-elasticsearch/src/test/java/org/apache/storm/elasticsearch/bolt/EsIndexBoltTest.java --- @@ -20,6 +20,7 @@ import backtype.storm.tuple.Tuple; import org.apache.storm.elasticsearch.common.EsConfig; import org.apache.storm.elasticsearch.common.EsTestUtil; +import org.apache.storm.elasticsearch.common.EsTupleMapper; import org.elasticsearch.action.count.CountRequest; import org.elasticsearch.action.count.CountRequestBuilder; import org.elasticsearch.action.count.CountResponse; --- End diff -- Line 24 25 27 are unnecessary imports too [storm-elasticsearch] Introduces Tuple - ES document mapper to get rid of constant field mapping (source, index, type) --- Key: STORM-974 URL: https://issues.apache.org/jira/browse/STORM-974 Project: Apache Storm Issue Type: Improvement Reporter: Jungtaek Lim Assignee: Jungtaek Lim For now EsIndexBolt uses constant field mapping (source, index, type) which is not flexible. We can introduce tuple - ES document mapper interface to let users define their relationship, as other external modules did. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-974) [storm-elasticsearch] Introduces Tuple - ES document mapper to get rid of constant field mapping (source, index, type)
[ https://issues.apache.org/jira/browse/STORM-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694625#comment-14694625 ] ASF GitHub Bot commented on STORM-974: -- Github user caofangkun commented on a diff in the pull request: https://github.com/apache/storm/pull/679#discussion_r36938034 --- Diff: external/storm-elasticsearch/src/main/java/org/apache/storm/elasticsearch/bolt/EsPercolateBolt.java --- @@ -31,14 +32,21 @@ --- End diff -- Trivial, this two line imports could be removed [storm-elasticsearch] Introduces Tuple - ES document mapper to get rid of constant field mapping (source, index, type) --- Key: STORM-974 URL: https://issues.apache.org/jira/browse/STORM-974 Project: Apache Storm Issue Type: Improvement Reporter: Jungtaek Lim Assignee: Jungtaek Lim For now EsIndexBolt uses constant field mapping (source, index, type) which is not flexible. We can introduce tuple - ES document mapper interface to let users define their relationship, as other external modules did. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-974 Introduces Tuple - ES document mapp...
Github user caofangkun commented on a diff in the pull request: https://github.com/apache/storm/pull/679#discussion_r36938144 --- Diff: external/storm-elasticsearch/src/main/java/org/apache/storm/elasticsearch/trident/EsStateFactory.java --- @@ -19,6 +19,7 @@ import backtype.storm.task.IMetricsContext; import org.apache.storm.elasticsearch.common.EsConfig; +import org.apache.storm.elasticsearch.common.EsTupleMapper; import org.slf4j.Logger; import org.slf4j.LoggerFactory; --- End diff -- this two line slf4j imports could be removed too --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-974 Introduces Tuple - ES document mapp...
Github user caofangkun commented on a diff in the pull request: https://github.com/apache/storm/pull/679#discussion_r36938292 --- Diff: external/storm-elasticsearch/src/test/java/org/apache/storm/elasticsearch/bolt/EsPercolateBoltTest.java --- @@ -21,6 +21,7 @@ import backtype.storm.tuple.Values; import org.apache.storm.elasticsearch.common.EsConfig; import org.apache.storm.elasticsearch.common.EsTestUtil; +import org.apache.storm.elasticsearch.common.EsTupleMapper; import org.elasticsearch.action.count.CountResponse; import org.elasticsearch.action.percolate.PercolateResponse; import org.elasticsearch.index.query.TermQueryBuilder; --- End diff -- Line 25 27 28 unneeded imports --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Assigned] (STORM-977) Incorrect signal (-9) when as-user is true
[ https://issues.apache.org/jira/browse/STORM-977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Liptak reassigned STORM-977: -- Assignee: Gabor Liptak Incorrect signal (-9) when as-user is true -- Key: STORM-977 URL: https://issues.apache.org/jira/browse/STORM-977 Project: Apache Storm Issue Type: Bug Affects Versions: 0.10.0 Reporter: Derek Dagit Assignee: Gabor Liptak Priority: Trivial Labels: Newbie https://github.com/apache/storm/blob/544e55cb8ab8878c4af500aab49bd35d4b69cd3e/storm-core/src/clj/backtype/storm/daemon/supervisor.clj#L265 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-988) supervisor.slots.ports should not allow duplicate element values
caofangkun created STORM-988: Summary: supervisor.slots.ports should not allow duplicate element values Key: STORM-988 URL: https://issues.apache.org/jira/browse/STORM-988 Project: Apache Storm Issue Type: Bug Reporter: caofangkun Assignee: caofangkun Priority: Trivial -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-974 Introduces Tuple - ES document mapp...
Github user caofangkun commented on a diff in the pull request: https://github.com/apache/storm/pull/679#discussion_r36938034 --- Diff: external/storm-elasticsearch/src/main/java/org/apache/storm/elasticsearch/bolt/EsPercolateBolt.java --- @@ -31,14 +32,21 @@ --- End diff -- Trivial, this two line imports could be removed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Update timer.clj
Github user wangli1426 commented on a diff in the pull request: https://github.com/apache/storm/pull/680#discussion_r36938783 --- Diff: storm-core/src/clj/backtype/storm/timer.clj --- @@ -53,8 +53,11 @@ ;; event generation. If any recurring events ;; are scheduled then we will always go ;; through this branch, sleeping only the - ;; exact necessary amount of time. - (Time/sleep (- time-millis (current-time-millis))) + ;; exact necessary amount of time. We give + ;; an upper bound, e.g. 1000 millis, to the + ;; sleeping time, to limit the response time + ;; for detecting any new event within 1 secs. + (Time/sleep (max 1000 (- time-millis (current-time-millis --- End diff -- Yes, you are right, min should be used definitely. Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-974) [storm-elasticsearch] Introduces Tuple - ES document mapper to get rid of constant field mapping (source, index, type)
[ https://issues.apache.org/jira/browse/STORM-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694638#comment-14694638 ] ASF GitHub Bot commented on STORM-974: -- Github user caofangkun commented on a diff in the pull request: https://github.com/apache/storm/pull/679#discussion_r36938292 --- Diff: external/storm-elasticsearch/src/test/java/org/apache/storm/elasticsearch/bolt/EsPercolateBoltTest.java --- @@ -21,6 +21,7 @@ import backtype.storm.tuple.Values; import org.apache.storm.elasticsearch.common.EsConfig; import org.apache.storm.elasticsearch.common.EsTestUtil; +import org.apache.storm.elasticsearch.common.EsTupleMapper; import org.elasticsearch.action.count.CountResponse; import org.elasticsearch.action.percolate.PercolateResponse; import org.elasticsearch.index.query.TermQueryBuilder; --- End diff -- Line 25 27 28 unneeded imports [storm-elasticsearch] Introduces Tuple - ES document mapper to get rid of constant field mapping (source, index, type) --- Key: STORM-974 URL: https://issues.apache.org/jira/browse/STORM-974 Project: Apache Storm Issue Type: Improvement Reporter: Jungtaek Lim Assignee: Jungtaek Lim For now EsIndexBolt uses constant field mapping (source, index, type) which is not flexible. We can introduce tuple - ES document mapper interface to let users define their relationship, as other external modules did. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-974 Introduces Tuple - ES document mapp...
Github user caofangkun commented on a diff in the pull request: https://github.com/apache/storm/pull/679#discussion_r36938233 --- Diff: external/storm-elasticsearch/src/test/java/org/apache/storm/elasticsearch/bolt/EsIndexBoltTest.java --- @@ -20,6 +20,7 @@ import backtype.storm.tuple.Tuple; import org.apache.storm.elasticsearch.common.EsConfig; import org.apache.storm.elasticsearch.common.EsTestUtil; +import org.apache.storm.elasticsearch.common.EsTupleMapper; import org.elasticsearch.action.count.CountRequest; import org.elasticsearch.action.count.CountRequestBuilder; import org.elasticsearch.action.count.CountResponse; --- End diff -- Line 24 25 27 are unnecessary imports too --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-974) [storm-elasticsearch] Introduces Tuple - ES document mapper to get rid of constant field mapping (source, index, type)
[ https://issues.apache.org/jira/browse/STORM-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694629#comment-14694629 ] ASF GitHub Bot commented on STORM-974: -- Github user caofangkun commented on a diff in the pull request: https://github.com/apache/storm/pull/679#discussion_r36938144 --- Diff: external/storm-elasticsearch/src/main/java/org/apache/storm/elasticsearch/trident/EsStateFactory.java --- @@ -19,6 +19,7 @@ import backtype.storm.task.IMetricsContext; import org.apache.storm.elasticsearch.common.EsConfig; +import org.apache.storm.elasticsearch.common.EsTupleMapper; import org.slf4j.Logger; import org.slf4j.LoggerFactory; --- End diff -- this two line slf4j imports could be removed too [storm-elasticsearch] Introduces Tuple - ES document mapper to get rid of constant field mapping (source, index, type) --- Key: STORM-974 URL: https://issues.apache.org/jira/browse/STORM-974 Project: Apache Storm Issue Type: Improvement Reporter: Jungtaek Lim Assignee: Jungtaek Lim For now EsIndexBolt uses constant field mapping (source, index, type) which is not flexible. We can introduce tuple - ES document mapper interface to let users define their relationship, as other external modules did. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-950) Apache Storm website redesign
[ https://issues.apache.org/jira/browse/STORM-950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694189#comment-14694189 ] ASF GitHub Bot commented on STORM-950: -- Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/670#issuecomment-130452561 @harshach I'm not suggesting we create a feature branch, I'm suggesting we change the way we publish the site to make it easier for people to contribute to. I forgot to mention it before, but I'm happy to set that up. Some of the issues I see are minor layout issues (twitter feed should be below news, for example), some are maintenance-related (the recent news on the home page should be generated, not hard-coded in HTML), and some are related to ASF policy (we are pulling in content from facebook.com and google.com domains that potentially includes end-user tracking, and I'm not sure if that's allowed -- I'd have to check). I'm not worried about docs/textual content, that can be fixed anytime. Apache Storm website redesign - Key: STORM-950 URL: https://issues.apache.org/jira/browse/STORM-950 Project: Apache Storm Issue Type: Improvement Reporter: Sriharsha Chintalapani Assignee: Sriharsha Chintalapani -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: Corrected ConnectionPrvoider to ConnectionProv...
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/678#issuecomment-130508217 @randerzander Yes, right. I'll take care of it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-950) Apache Storm website redesign
[ https://issues.apache.org/jira/browse/STORM-950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694154#comment-14694154 ] ASF GitHub Bot commented on STORM-950: -- Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/670#issuecomment-130444965 I have a few things that I think need to be fixed before we publish this. But rather than enumerate every last little thing and ask you to fix them, I'd rather jump in and fix them myself and let others do the same. I propose we move this off to an empty branch that just contains the storm website code -- similar to the way github pages work. Then anyone could help fix things by opening a pull request against that branch. Then once we are all satisfied enough to publish it, we do so and remove the website code from the main source branches. I don't have a reference handy, but I think ASF INFRA has a way to publish web content from a specially-named branch (similar to how gh-pages branches work). That would eliminate the extra step of generating the site to SVN and committing that way. I'll look into this option some more. How does that sound? Apache Storm website redesign - Key: STORM-950 URL: https://issues.apache.org/jira/browse/STORM-950 Project: Apache Storm Issue Type: Improvement Reporter: Sriharsha Chintalapani Assignee: Sriharsha Chintalapani -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-988: supervisor.slots.ports should not a...
GitHub user caofangkun opened a pull request: https://github.com/apache/storm/pull/681 STORM-988: supervisor.slots.ports should not allow duplicate element values You can merge this pull request into a Git repository by running: $ git pull https://github.com/caofangkun/apache-storm storm-988 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/681.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #681 commit 26ea627fb39fb84f31de773e650318a81eba8bd4 Author: caofangkun caofang...@gmail.com Date: 2015-08-13T02:41:23Z STORM-988: supervisor.slots.ports should not allow duplicate element values --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Corrected ConnectionPrvoider to ConnectionProv...
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/678#issuecomment-130212024 Seems like something is messed. - https://github.com/apache/storm/pull/556 - https://github.com/Parth-Brahmbhatt/incubator-storm/commit/d268903f026cf275b72e6246b61c769d28fb3ed1 It was already fixed. Is it missed spot, or messed thing? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Corrected ConnectionPrvoider to ConnectionProv...
GitHub user randerzander opened a pull request: https://github.com/apache/storm/pull/678 Corrected ConnectionPrvoider to ConnectionProvider You can merge this pull request into a Git repository by running: $ git pull https://github.com/randerzander/storm master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/678.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #678 commit 26dc0390dd7d14c1f8cf4f834202ec778e5d82c0 Author: Randy Gelhausen rgel...@gmail.com Date: 2015-08-12T06:36:24Z Corrected ConnectionPrvoider to ConnectionProvider --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Corrected ConnectionPrvoider to ConnectionProv...
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/678#issuecomment-130222094 My bad. Sorry, it seems to be just missed spot. ;) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Corrected ConnectionPrvoider to ConnectionProv...
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/678#issuecomment-130223541 +1. Since it modifies method name, it should be applied to 0.10.0 to keep backward compatibility with 0.10.0 and 0.11.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Corrected ConnectionPrvoider to ConnectionProv...
Github user randerzander commented on the pull request: https://github.com/apache/storm/pull/678#issuecomment-130236052 How do you want to handle the .10 update? Separate PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-594) Auto-Scaling Resources in a Topology
[ https://issues.apache.org/jira/browse/STORM-594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14693732#comment-14693732 ] Boyang Jerry Peng commented on STORM-594: - Hello guys, Recently there has been paper's published in academia that use existing proofs in the area of queueing theory to calculate the parallelism of each component. Autoscaling or elasticity in storm can be viewed as a queuing theory problem since in storm tuples need to be processed and there are only a limited number of processor's to process them. Auto-Scaling Resources in a Topology Key: STORM-594 URL: https://issues.apache.org/jira/browse/STORM-594 Project: Apache Storm Issue Type: New Feature Reporter: HARSHA BALASUBRAMANIAN Assignee: Pooyan Jamshidi Priority: Minor Attachments: Algorithm for Auto-Scaling.pdf, Project Plan and Scope.pdf Original Estimate: 504h Remaining Estimate: 504h A useful feature missing in Storm topologies is the ability to auto-scale resources, based on a pre-configured metric. The feature proposed here aims to build such a auto-scaling mechanism using a feedback system. A brief overview of the feature is provided here. The finer details of the required components and the scaling algorithm (uses a Feedback System) are provided in the PDFs attached. Brief Overview: Topologies may get created with or (ideally) without parallelism hints and tasks in their bolts and spouts, before submitting them, If auto-scaling is set in the topology (using a Boolean flag), the topology will also get submitted to the auto-scale module. The auto-scale module will read a pre-configured metric (threshold/min) from a configuration file. Using this value, the topology's resources will be modified till the threshold is reached. At each stage in the auto-scale module's execution, feedback from the previous execution will be used to tune the resources. The systems that need to be in place to achieve this are: 1. Metrics which provide the current threshold (no: of acks per minute) for a topology's spouts and bolts. 2. Access to Storm's CLI tool which can change a topology's resources are runtime. 3. A new java or clojure module which runs within the Nimbus daemon or in parallel to it. This will be the auto-scale module. Limitations: (This is not an exhaustive list. More will be added as the design matures. Also, some of the points here may get resolved) To test the feature there will be a number of limitations in the first release. As the feature matures, it will be allowed to scale more 1. The auto-scale module will be limited to a few topologies (maybe 4 or 5 at maximum) 2. New bolts will not be added to scale a topology. This feature will be limited to increasing the resources within the existing topology. 3. Topology resources will not be decreased when it is running at more than the required number (except for a few cases) 4. This feature will work only for long-running topologies where the input threshold can become equal to or greater than the required threshold -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-974) [storm-elasticsearch] Introduces Tuple - ES document mapper to get rid of constant field mapping (source, index, type)
[ https://issues.apache.org/jira/browse/STORM-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14693503#comment-14693503 ] ASF GitHub Bot commented on STORM-974: -- GitHub user HeartSaVioR opened a pull request: https://github.com/apache/storm/pull/679 STORM-974 Introduces Tuple - ES document mapper to get rid of constant field mapping * EsTupleMapper defines how to extract source, index, type, and id from tuple for ElasticSearch. * Applies EsTupleMapper to completely get rid of constant field mapping. * Also modifying README.md to show how to use. You can merge this pull request into a Git repository by running: $ git pull https://github.com/HeartSaVioR/storm STORM-974 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/679.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #679 commit 1f93a3f0f8194715c35869766dec5fc623066d3e Author: Jungtaek Lim kabh...@gmail.com Date: 2015-08-12T13:46:13Z STORM-974 Introduces Tuple - ES document mapper to get rid of constant field mapping * EsTupleMapper defines how to extract source, index, type, and id from tuple for ElasticSearch. * Applies EsTupleMapper to completely get rid of constant field mapping. * Also modifying README.md to show how to use. [storm-elasticsearch] Introduces Tuple - ES document mapper to get rid of constant field mapping (source, index, type) --- Key: STORM-974 URL: https://issues.apache.org/jira/browse/STORM-974 Project: Apache Storm Issue Type: Improvement Reporter: Jungtaek Lim For now EsIndexBolt uses constant field mapping (source, index, type) which is not flexible. We can introduce tuple - ES document mapper interface to let users define their relationship, as other external modules did. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-837) HdfsState ignores commits
[ https://issues.apache.org/jira/browse/STORM-837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14693521#comment-14693521 ] ASF GitHub Bot commented on STORM-837: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/644#issuecomment-130315702 For me internally the tests pass and everything builds. Although the rest of storm seems a bit unstable with tests. I think the changes look fine +1 for merging this in. I don't think we have any split brain problems because of how we route messages, so the begin command and commit messages should prevent this. There are some things that I think we can do better, but are not wrong. 1) I am scared that the transaction file is going to put more of a load on the Name Node than I like. I am mostly concerned that if we have thousands of instances of hdfs state creating/deleting transaction files multiple times a second the load is going to noticeable on a large cluster. But this is a premature optimization so I would say we don't address it unless we see it as an actual problem. 2) The RotationActions don't have a recovery interface, and things like the MoveFile action could potentially fail after successfully rotating the file which might leave the file in the wrong directory. 3) I don't really like the special case for the TimedRotationAction.start. I would rather see start a part of the RotationAction interface, that can be ignored by those that don't want/need it. The last two could perhaps be a follow on JIRA. I am fine with the code as is, but I would like another pair of eyes looking at this too before merging it in. HdfsState ignores commits - Key: STORM-837 URL: https://issues.apache.org/jira/browse/STORM-837 Project: Apache Storm Issue Type: Bug Reporter: Robert Joseph Evans Assignee: Arun Mahadevan Priority: Critical HdfsState works with trident which is supposed to provide exactly once processing. It does this two ways, first by informing the state about commits so it can be sure the data is written out, and second by having a commit id, so that double commits can be handled. HdfsState ignores the beginCommit and commit calls, and with that ignores the ids. This means that if you use HdfsState and your worker crashes you may both lose data and get some data twice. At a minimum the flush and file rotation should be tied to the commit in some way. The commit ID should at a minimum be written out with the data so someone reading the data can have a hope of deduping it themselves. Also with the rotationActions it is possible for a file that was partially written is leaked, and never moved to the final location, because it is not rotated. I personally think the actions are too generic for this case and need to be deprecated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-837] Support for exactly once semantics...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/644#issuecomment-130315702 For me internally the tests pass and everything builds. Although the rest of storm seems a bit unstable with tests. I think the changes look fine +1 for merging this in. I don't think we have any split brain problems because of how we route messages, so the begin command and commit messages should prevent this. There are some things that I think we can do better, but are not wrong. 1) I am scared that the transaction file is going to put more of a load on the Name Node than I like. I am mostly concerned that if we have thousands of instances of hdfs state creating/deleting transaction files multiple times a second the load is going to noticeable on a large cluster. But this is a premature optimization so I would say we don't address it unless we see it as an actual problem. 2) The RotationActions don't have a recovery interface, and things like the MoveFile action could potentially fail after successfully rotating the file which might leave the file in the wrong directory. 3) I don't really like the special case for the TimedRotationAction.start. I would rather see start a part of the RotationAction interface, that can be ignored by those that don't want/need it. The last two could perhaps be a follow on JIRA. I am fine with the code as is, but I would like another pair of eyes looking at this too before merging it in. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-851) Storm Solr connector
[ https://issues.apache.org/jira/browse/STORM-851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14693536#comment-14693536 ] ASF GitHub Bot commented on STORM-851: -- Github user harshach commented on a diff in the pull request: https://github.com/apache/storm/pull/665#discussion_r36863654 --- Diff: external/storm-solr/src/main/java/org/apache/storm/solr/bolt/AbstractSolrBolt.java --- @@ -0,0 +1,33 @@ +package org.apache.storm.solr.bolt; + +import backtype.storm.task.OutputCollector; +import backtype.storm.task.TopologyContext; +import backtype.storm.topology.OutputFieldsDeclarer; +import backtype.storm.topology.base.BaseRichBolt; +import backtype.storm.tuple.Tuple; +import org.apache.solr.client.solrj.SolrClient; +import org.apache.solr.client.solrj.impl.CloudSolrClient; +import org.apache.storm.solr.config.SolrConfig; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.Map; + +/** + * Created by hlouro on 7/17/15. --- End diff -- can you remove these comments. Storm Solr connector Key: STORM-851 URL: https://issues.apache.org/jira/browse/STORM-851 Project: Apache Storm Issue Type: Improvement Reporter: Sriharsha Chintalapani Assignee: Hugo Louro Storm solr connector should provide bolt and trident implementation to allow users to index data coming through the topology into solr. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-851: Storm Solr Connector
Github user harshach commented on a diff in the pull request: https://github.com/apache/storm/pull/665#discussion_r36863654 --- Diff: external/storm-solr/src/main/java/org/apache/storm/solr/bolt/AbstractSolrBolt.java --- @@ -0,0 +1,33 @@ +package org.apache.storm.solr.bolt; + +import backtype.storm.task.OutputCollector; +import backtype.storm.task.TopologyContext; +import backtype.storm.topology.OutputFieldsDeclarer; +import backtype.storm.topology.base.BaseRichBolt; +import backtype.storm.tuple.Tuple; +import org.apache.solr.client.solrj.SolrClient; +import org.apache.solr.client.solrj.impl.CloudSolrClient; +import org.apache.storm.solr.config.SolrConfig; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.Map; + +/** + * Created by hlouro on 7/17/15. --- End diff -- can you remove these comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Update timer.clj
GitHub user wangli1426 opened a pull request: https://github.com/apache/storm/pull/680 Update timer.clj Issue description: The timer thread calculates the delay to schedule the head of the queue and sleeps accordingly. However, if a new event with high priority is inserted at the head of the queue during the sleep, this event cannot be scheduled in time. Solution: Give a upper bound to the sleeping time, to guarantee a bounded response time for detecting new events. You can merge this pull request into a Git repository by running: $ git pull https://github.com/wangli1426/storm master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/680.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #680 commit d1a6f4a753661f7e825eeec0c51782462fca4d5f Author: Li Wang wangli1...@gmail.com Date: 2015-08-12T14:00:15Z Update timer.clj Give a upper bound to the sleeping time, to guarantee a bounded response time for detecting new events. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-974 Introduces Tuple - ES document mapp...
GitHub user HeartSaVioR opened a pull request: https://github.com/apache/storm/pull/679 STORM-974 Introduces Tuple - ES document mapper to get rid of constant field mapping * EsTupleMapper defines how to extract source, index, type, and id from tuple for ElasticSearch. * Applies EsTupleMapper to completely get rid of constant field mapping. * Also modifying README.md to show how to use. You can merge this pull request into a Git repository by running: $ git pull https://github.com/HeartSaVioR/storm STORM-974 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/679.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #679 commit 1f93a3f0f8194715c35869766dec5fc623066d3e Author: Jungtaek Lim kabh...@gmail.com Date: 2015-08-12T13:46:13Z STORM-974 Introduces Tuple - ES document mapper to get rid of constant field mapping * EsTupleMapper defines how to extract source, index, type, and id from tuple for ElasticSearch. * Applies EsTupleMapper to completely get rid of constant field mapping. * Also modifying README.md to show how to use. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: How does external module storm-jdbc implement transactional
Yes JDBC state has the same problem that I found in HDFS state recently. It is ignoring the commits. https://github.com/apache/storm/blob/master/external/storm-jdbc/src/main/java/org/apache/storm/jdbc/trident/state/JdbcState.java#L114-L122 Please file a JIRA to update JdbcState to make it transactional. Depending on the size of the transactions we probably should just use the database's build in transactions. - Bobby On Tuesday, August 11, 2015 8:57 PM, ght230 ght...@163.com wrote: Hello : I had a problem during using the module storm/external/storm-jdbc, the class JdbcState in this module does not seem to distinguish between opaque transactional state, transactional state, and non-transactional state, how can it implement the transaction executed only once? This function is forgotten, or any other reason? Thanks! iceguo
[GitHub] storm pull request: STORM-980 Re-include storm-kafka to Travis CI ...
Github user d2r commented on the pull request: https://github.com/apache/storm/pull/674#issuecomment-130387500 +1 on your testing of this. We can revert if it does not work. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-980) Re-include storm-kafka tests from Travis CI build
[ https://issues.apache.org/jira/browse/STORM-980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14693922#comment-14693922 ] ASF GitHub Bot commented on STORM-980: -- Github user d2r commented on the pull request: https://github.com/apache/storm/pull/674#issuecomment-130387500 +1 on your testing of this. We can revert if it does not work. Re-include storm-kafka tests from Travis CI build - Key: STORM-980 URL: https://issues.apache.org/jira/browse/STORM-980 Project: Apache Storm Issue Type: Sub-task Components: storm-kafka Reporter: Jungtaek Lim Assignee: Jungtaek Lim Recently [~zhuoliu] reported that randome unit tests failures on storm-kafka may be resolved with STORM-826. I rebuilt test-kafka-test branch 5 times in a row (tested with jdk7, jdk8, so storm-kafka unit tests were run 10 times), and I can see there's no longer storm-kafka's random failure. It is broken at 6th trial, but I've met random failure of storm-core first, so we're leaving decision whether we ignore low probabilities of random failures or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-798) storm-kafka tests are failing intermittently
[
https://issues.apache.org/jira/browse/STORM-798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14693934#comment-14693934
]
Derek Dagit commented on STORM-798:
---
Linking the JIRA that probably fixed this issue.
The Sub-Tasks are closed now. If the CI build succeeds without storm-kafka
tests failing, we should be able to close this Bug.
storm-kafka tests are failing intermittently
Key: STORM-798
URL: https://issues.apache.org/jira/browse/STORM-798
Project: Apache Storm
Issue Type: Bug
Components: storm-kafka
Reporter: Jungtaek Lim
Priority: Minor
I'm trying to adopt Travis CI and observed failures about storm-kafka test.
https://travis-ci.org/apache/storm/jobs/59795630
Full log is here.
https://s3.amazonaws.com/archive.travis-ci.org/jobs/59795629/log.txt
{noformat}
Tests in error:
testKeyValue(storm.kafka.TridentKafkaTest): Could not send message with key
= key-123 to topic = test
generateTuplesWithKeyAndKeyValueScheme(storm.kafka.KafkaUtilsTest): Error
fetching data from [Partition{host=localhost:52047, partition=0}] for topic
[testTopic]: [OFFSET_OUT_OF_RANGE]
{noformat}
Please note that tests run with VM, which could be slow at moment or while
testing.
It would be better to let tests pass from slow box without random failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: Corrected ConnectionPrvoider to ConnectionProv...
Github user randerzander commented on the pull request: https://github.com/apache/storm/pull/678#issuecomment-130415963 that's something you need to be comitter to do, right? I'm not sure how to use one PR to commit to multiple branches. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-975] Storm-Kafka trident topology examp...
Github user harshach commented on the pull request: https://github.com/apache/storm/pull/666#issuecomment-130404466 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (STORM-987) Document how to recover TopicOffsetOutOfRange exception in Storm-Kafka Connector
Sriharsha Chintalapani created STORM-987: Summary: Document how to recover TopicOffsetOutOfRange exception in Storm-Kafka Connector Key: STORM-987 URL: https://issues.apache.org/jira/browse/STORM-987 Project: Apache Storm Issue Type: Documentation Reporter: Sriharsha Chintalapani Assignee: Parth Brahmbhatt KafkaSpout can land in TopicOffsetOutOfRangeException: Got fetch request with offset out of range . We need to document how to recover from this case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-980 Re-include storm-kafka to Travis CI ...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/674 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-893) Resource Aware Scheduling
[ https://issues.apache.org/jira/browse/STORM-893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14693875#comment-14693875 ] Boyang Jerry Peng commented on STORM-893: - Through collaboration with people at yahoo, I have published a paper in the Middleware 2015 conference on resource aware scheduling. The link to the paper is listed below: http://web.engr.illinois.edu/~bpeng/files/r-storm.pdf This paper hasn't been officially published since the conference is held in December, but the paper has been accepted. I was a student at UIUC but now I am working at Yahoo. We are currently actively working on get a implementation of the resource aware scheduler to be used in production. I will also be working on a open source version, that will hopefully get pushed back into the community soon. Resource Aware Scheduling - Key: STORM-893 URL: https://issues.apache.org/jira/browse/STORM-893 Project: Apache Storm Issue Type: Umbrella Reporter: Robert Joseph Evans Assignee: Boyang Jerry Peng At Yahoo we have been working on resource aware scheduling in storm, based off of some work done in academia. This rollup ticket is to track the complete project. With several sub tasks. Some that are already done and need to be pushed back, and others that we have not started on yet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (STORM-983) If stormconf.ser is not downloaded correctly, the supervisor will bounce for ever
[ https://issues.apache.org/jira/browse/STORM-983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng reassigned STORM-983: --- Assignee: Boyang Jerry Peng If stormconf.ser is not downloaded correctly, the supervisor will bounce for ever - Key: STORM-983 URL: https://issues.apache.org/jira/browse/STORM-983 Project: Apache Storm Issue Type: Bug Reporter: Reza Farivar Assignee: Boyang Jerry Peng Priority: Minor If the stormconf.ser is corrupted, and for whatever reason exists with size 0, the following function returns with an EOF exception, which crashes the Supervisor. Upon restart, supervisor tries to read it again, which results in another crash, etc. https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/utils/Utils.java#L155 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
