[jira] [Commented] (STORM-671) Measure tuple serialization/deserialization latency.
[ https://issues.apache.org/jira/browse/STORM-671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395749#comment-14395749 ] ASF GitHub Bot commented on STORM-671: -- Github user Lewuathe commented on the pull request: https://github.com/apache/storm/pull/442#issuecomment-89586916 @revans2 Thank you for supporting. I'll change this metrics as optional and thrift interface. So all I have to do it 2 points, @Parth-Brahmbhatt ? If there is any other points I should care about, please let me know. * Store metrics in thrift in ZooKeeper. * Change serialization as optional metrics. Measure tuple serialization/deserialization latency. Key: STORM-671 URL: https://issues.apache.org/jira/browse/STORM-671 Project: Apache Storm Issue Type: New Feature Reporter: Robert Joseph Evans Assignee: Kai Sasaki Some times the serialization/deserialization cost can be very high, and it is not currently measured anywhere in storm. We should measure it, at least in a similar way to how we do execute and process latency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-671] Measure tuple serialization/deseri...
Github user Lewuathe commented on the pull request: https://github.com/apache/storm/pull/442#issuecomment-89586916 @revans2 Thank you for supporting. I'll change this metrics as optional and thrift interface. So all I have to do it 2 points, @Parth-Brahmbhatt ? If there is any other points I should care about, please let me know. * Store metrics in thrift in ZooKeeper. * Change serialization as optional metrics. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-650) Storm-Kafka Refactoring and Improvements
[ https://issues.apache.org/jira/browse/STORM-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395777#comment-14395777 ] Jay Kreps commented on STORM-650: - Hey [~wurstmeister] and [~ptgoetz] we are going to release that Kafka consumer API in the next release. I believe it should cover the needs you guys have and should dramatically simplify the kafka-storm code--it internally handles server failure, partition migration, offset storage, etc, but gives you full control over partition assignment and offset commit points which are the needs of a stream processing system. This should also remove all unnecessary threading from your code too as the consumer is fully non-blocking. Prior to the Kafka release it would be great if someone who knows the storm-kafka integration well could do a deep dive and just validate that these apis would indeed cover your needs and also validate that it would really significantly simplify your life. We think both should be true, but it would be good to check so we can make changes if needed. Once it is released we will have to break compatibility with each change so flushing these things out now is just much easier. I'd be happy to jump on a quick call to discuss if that is useful. Storm-Kafka Refactoring and Improvements Key: STORM-650 URL: https://issues.apache.org/jira/browse/STORM-650 Project: Apache Storm Issue Type: Improvement Components: storm-kafka Reporter: P. Taylor Goetz This is intended to be a parent/umbrella JIRA covering a number of efforts/suggestions aimed at improving the storm-kafka module. The goal is to facilitate communication and collaboration by providing a central point for discussion and coordination. The first phase should be to identify and agree upon a list of high-level points we would like to address. Once that is complete, we can move on to implementation/design discussions, followed by an implementation plan, division of labor, etc. A non-exhaustive, initial list of items follows. New/additional thoughts can be proposed in the comments. * Improve API for Specifying the Kafka Starting point Configuring the kafka spout's starting position (e.g. forceFromStart=true) is a common source of confusion. This should be refactored to provide an easy to understand, unambiguous API for configuring this property. * Use Kafka APIs Instead of Internal ZK Metadata (STORM-590) Currently the Kafka spout relies on reading Kafka's internal metadata from zookeeper. This should be refactored to use the Kafka Consumer API to protect against changes to the internal metadata format stored in ZK. * Improve Error Handling There are a number of failure scenarios with the kafka spout that users may want to react to differently based on their use case. Add a failure handler API that allows users to implement and/or plug in alternative failure handling implementations. It is assumed that default (sane) implementations would be included and configured by default. * Configuration/General Refactoring (BrokerHosts, etc.) (STORM-631) (need to flesh this out better) Reduce unnecessary marker interfaces/instance of checks. Unify configuration of core storm/trident spout implementations. * Kafka Spout doesn't pick up from the beginning of the queue unless forceFromStart specified (STORM-563) Discussion Items: * How important is backward compatibility? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: An option for new/updated Redis keys to expire...
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/501#issuecomment-89695498 You can apply your changeset to RedisClusterMapState though it can't use Pipeline for performance. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Comment Edited] (STORM-753) [storm-redis] Let Redis*StateUpdater to have additional mapper - converting Redis key/value to Storm Values
[ https://issues.apache.org/jira/browse/STORM-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14396013#comment-14396013 ] Jungtaek Lim edited comment on STORM-753 at 4/5/15 3:23 AM: We can use RedisLookupMapper / RedisStoreMapper with restricting data type to STRING and HASH. was (Author: kabhwan): We can use RedisLookupMapper / RedisStoreMapper with restricting data type to STRING. Maybe we can use HASH too, but it should be along with RedisMapState / RedisClusterMapState. I'll file a new issue regarding it. [storm-redis] Let Redis*StateUpdater to have additional mapper - converting Redis key/value to Storm Values --- Key: STORM-753 URL: https://issues.apache.org/jira/browse/STORM-753 Project: Apache Storm Issue Type: Improvement Affects Versions: 0.10.0, 0.11.0 Reporter: Jungtaek Lim Assignee: Jungtaek Lim https://issues.apache.org/jira/browse/STORM-723?focusedCommentId=14381626page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14381626 Currently Redis*StateUpdater use key prefix to let users customize key name. Since it's less powerful, how about having 2 mappers which one is for converting tuple to key/value, and another one is for converting value from Redis to Storm Values? You can refer https://github.com/apache/storm/blob/master/external/storm-hbase/src/main/java/org/apache/storm/hbase/trident/state/HBaseState.java. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-753] Improve RedisStateQuerier to conve...
Github user HeartSaVioR closed the pull request at: https://github.com/apache/storm/pull/503 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-753) [storm-redis] Let Redis*StateUpdater to have additional mapper - converting Redis key/value to Storm Values
[ https://issues.apache.org/jira/browse/STORM-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14396047#comment-14396047 ] ASF GitHub Bot commented on STORM-753: -- Github user HeartSaVioR closed the pull request at: https://github.com/apache/storm/pull/503 [storm-redis] Let Redis*StateUpdater to have additional mapper - converting Redis key/value to Storm Values --- Key: STORM-753 URL: https://issues.apache.org/jira/browse/STORM-753 Project: Apache Storm Issue Type: Improvement Affects Versions: 0.10.0, 0.11.0 Reporter: Jungtaek Lim Assignee: Jungtaek Lim https://issues.apache.org/jira/browse/STORM-723?focusedCommentId=14381626page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14381626 Currently Redis*StateUpdater use key prefix to let users customize key name. Since it's less powerful, how about having 2 mappers which one is for converting tuple to key/value, and another one is for converting value from Redis to Storm Values? You can refer https://github.com/apache/storm/blob/master/external/storm-hbase/src/main/java/org/apache/storm/hbase/trident/state/HBaseState.java. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-753) [storm-redis] Let Redis*StateUpdater to have additional mapper - converting Redis key/value to Storm Values
[ https://issues.apache.org/jira/browse/STORM-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14396014#comment-14396014 ] ASF GitHub Bot commented on STORM-753: -- GitHub user HeartSaVioR opened a pull request: https://github.com/apache/storm/pull/503 Improve RedisStateQuerier to convert ListValues from Redis value Please see https://issues.apache.org/jira/browse/STORM-753 for more details. You can merge this pull request into a Git repository by running: $ git pull https://github.com/HeartSaVioR/storm STORM-753 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/503.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #503 commit 7fec9a1f4b9887897c9c7d0f3f80433fa34f18c0 Author: Jungtaek Lim kabh...@gmail.com Date: 2015-04-05T00:38:54Z Improve RedisStateQuerier to convert ListValues from Redis value * Reuse RedisStoreMapper / RedisLookupMapper ** RedisState*Querier / RedisState*Updater [storm-redis] Let Redis*StateUpdater to have additional mapper - converting Redis key/value to Storm Values --- Key: STORM-753 URL: https://issues.apache.org/jira/browse/STORM-753 Project: Apache Storm Issue Type: Improvement Affects Versions: 0.10.0, 0.11.0 Reporter: Jungtaek Lim Assignee: Jungtaek Lim https://issues.apache.org/jira/browse/STORM-723?focusedCommentId=14381626page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14381626 Currently Redis*StateUpdater use key prefix to let users customize key name. Since it's less powerful, how about having 2 mappers which one is for converting tuple to key/value, and another one is for converting value from Redis to Storm Values? You can refer https://github.com/apache/storm/blob/master/external/storm-hbase/src/main/java/org/apache/storm/hbase/trident/state/HBaseState.java. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-753) [storm-redis] Let Redis*StateUpdater to have additional mapper - converting Redis key/value to Storm Values
[ https://issues.apache.org/jira/browse/STORM-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14396013#comment-14396013 ] Jungtaek Lim commented on STORM-753: We can use RedisLookupMapper / RedisStoreMapper with restricting data type to STRING. Maybe we can use HASH too, but it should be along with RedisMapState / RedisClusterMapState. I'll file a new issue regarding it. [storm-redis] Let Redis*StateUpdater to have additional mapper - converting Redis key/value to Storm Values --- Key: STORM-753 URL: https://issues.apache.org/jira/browse/STORM-753 Project: Apache Storm Issue Type: Improvement Affects Versions: 0.10.0, 0.11.0 Reporter: Jungtaek Lim Assignee: Jungtaek Lim https://issues.apache.org/jira/browse/STORM-723?focusedCommentId=14381626page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14381626 Currently Redis*StateUpdater use key prefix to let users customize key name. Since it's less powerful, how about having 2 mappers which one is for converting tuple to key/value, and another one is for converting value from Redis to Storm Values? You can refer https://github.com/apache/storm/blob/master/external/storm-hbase/src/main/java/org/apache/storm/hbase/trident/state/HBaseState.java. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: Improve RedisStateQuerier to convert ListValu...
GitHub user HeartSaVioR opened a pull request: https://github.com/apache/storm/pull/503 Improve RedisStateQuerier to convert ListValues from Redis value Please see https://issues.apache.org/jira/browse/STORM-753 for more details. You can merge this pull request into a Git repository by running: $ git pull https://github.com/HeartSaVioR/storm STORM-753 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/503.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #503 commit 7fec9a1f4b9887897c9c7d0f3f80433fa34f18c0 Author: Jungtaek Lim kabh...@gmail.com Date: 2015-04-05T00:38:54Z Improve RedisStateQuerier to convert ListValues from Redis value * Reuse RedisStoreMapper / RedisLookupMapper ** RedisState*Querier / RedisState*Updater --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-594) Auto-Scaling Resources in a Topology
[ https://issues.apache.org/jira/browse/STORM-594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14396029#comment-14396029 ] Rohit Sanbhadti commented on STORM-594: --- Hi, We're a group of three CS students at UIUC and we have been working on this as a research project. We already have some initial implementation and are in the process of trying out other algorithms besides those suggested in the attached PDFs. Auto-Scaling Resources in a Topology Key: STORM-594 URL: https://issues.apache.org/jira/browse/STORM-594 Project: Apache Storm Issue Type: New Feature Reporter: HARSHA BALASUBRAMANIAN Assignee: SHAILESH PILARE Priority: Minor Attachments: Algorithm for Auto-Scaling.pdf, Project Plan and Scope.pdf Original Estimate: 504h Remaining Estimate: 504h A useful feature missing in Storm topologies is the ability to auto-scale resources, based on a pre-configured metric. The feature proposed here aims to build such a auto-scaling mechanism using a feedback system. A brief overview of the feature is provided here. The finer details of the required components and the scaling algorithm (uses a Feedback System) are provided in the PDFs attached. Brief Overview: Topologies may get created with or (ideally) without parallelism hints and tasks in their bolts and spouts, before submitting them, If auto-scaling is set in the topology (using a Boolean flag), the topology will also get submitted to the auto-scale module. The auto-scale module will read a pre-configured metric (threshold/min) from a configuration file. Using this value, the topology's resources will be modified till the threshold is reached. At each stage in the auto-scale module's execution, feedback from the previous execution will be used to tune the resources. The systems that need to be in place to achieve this are: 1. Metrics which provide the current threshold (no: of acks per minute) for a topology's spouts and bolts. 2. Access to Storm's CLI tool which can change a topology's resources are runtime. 3. A new java or clojure module which runs within the Nimbus daemon or in parallel to it. This will be the auto-scale module. Limitations: (This is not an exhaustive list. More will be added as the design matures. Also, some of the points here may get resolved) To test the feature there will be a number of limitations in the first release. As the feature matures, it will be allowed to scale more 1. The auto-scale module will be limited to a few topologies (maybe 4 or 5 at maximum) 2. New bolts will not be added to scale a topology. This feature will be limited to increasing the resources within the existing topology. 3. Topology resources will not be decreased when it is running at more than the required number (except for a few cases) 4. This feature will work only for long-running topologies where the input threshold can become equal to or greater than the required threshold -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-754) Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:java (default-cli) on project fyp-storm-try: The parameters 'mainClass' for goal org.codehaus.mojo:exec-ma
Lim Chun Yuen created STORM-754: --- Summary: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:java (default-cli) on project fyp-storm-try: The parameters 'mainClass' for goal org.codehaus.mojo:exec-maven-plugin:1.2.1:java are missing or invalid - [Help 1] Key: STORM-754 URL: https://issues.apache.org/jira/browse/STORM-754 Project: Apache Storm Issue Type: Question Affects Versions: 0.9.3 Environment: Ubuntu 14.04.1 LTS (GNU/Linux 3.13.0-44-generic x86_64) at Amazon Web Services EC2 Reporter: Lim Chun Yuen Fix For: 0.9.3 ubuntu@ip-10-0-0-101:~/storm/examples/fyp-storm-try$ sudo mvn compile exec:java -D storm.topology=fyp.storm.try.Topology [INFO] Scanning for projects... [WARNING] [WARNING] Some problems were encountered while building the effective model for org.apache.storm:fyp-storm-try:jar:0.9.4 [WARNING] 'reporting.plugins.plugin.version' for org.apache.maven.plugins:maven-javadoc-plugin is missing. @ org.apache.storm:storm:0.9.4, /root/.m2/repository/org/apache/storm/storm/0.9.4/storm-0.9.4.pom, line 694, column 21 [WARNING] 'reporting.plugins.plugin.version' for org.apache.maven.plugins:maven-surefire-report-plugin is missing. @ org.apache.storm:storm:0.9.4, /root/.m2/repository/org/apache/storm/storm/0.9.4/storm-0.9.4.pom, line 660, column 21 [WARNING] [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build. [WARNING] [WARNING] For this reason, future Maven versions might no longer support building such malformed projects. [WARNING] [INFO] [INFO] [INFO] Building fyp-storm-try 0.9.4 [INFO] [INFO] [INFO] --- maven-remote-resources-plugin:1.2.1:process (default) @ fyp-storm-try --- [INFO] [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ fyp-storm-try --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/ubuntu/storm/examples/fyp-storm-try/multilang [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ fyp-storm-try --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 5 source files to /home/ubuntu/storm/examples/fyp-storm-try/target/classes [INFO] [INFO] --- clojure-maven-plugin:1.3.18:compile (compile) @ fyp-storm-try --- [INFO] [INFO] exec-maven-plugin:1.2.1:java (default-cli) @ fyp-storm-try [INFO] [INFO] exec-maven-plugin:1.2.1:java (default-cli) @ fyp-storm-try [INFO] [INFO] --- exec-maven-plugin:1.2.1:java (default-cli) @ fyp-storm-try --- [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 10.364s [INFO] Finished at: Wed Apr 01 12:43:59 UTC 2015 [INFO] Final Memory: 19M/51M [INFO] [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:java (default-cli) on project fyp-storm-try: The parameters 'mainClass' for goal org.codehaus.mojo:exec-maven-plugin:1.2.1:java are missing or invalid - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginParameterException -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-753) [storm-redis] Let Redis*StateUpdater to have additional mapper - converting Redis key/value to Storm Values
[ https://issues.apache.org/jira/browse/STORM-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14396052#comment-14396052 ] ASF GitHub Bot commented on STORM-753: -- Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/503#issuecomment-89710830 I added a commit to branch, but PR is not reflected. I'll try to post new PR. [storm-redis] Let Redis*StateUpdater to have additional mapper - converting Redis key/value to Storm Values --- Key: STORM-753 URL: https://issues.apache.org/jira/browse/STORM-753 Project: Apache Storm Issue Type: Improvement Affects Versions: 0.10.0, 0.11.0 Reporter: Jungtaek Lim Assignee: Jungtaek Lim https://issues.apache.org/jira/browse/STORM-723?focusedCommentId=14381626page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14381626 Currently Redis*StateUpdater use key prefix to let users customize key name. Since it's less powerful, how about having 2 mappers which one is for converting tuple to key/value, and another one is for converting value from Redis to Storm Values? You can refer https://github.com/apache/storm/blob/master/external/storm-hbase/src/main/java/org/apache/storm/hbase/trident/state/HBaseState.java. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-753] Improve RedisStateQuerier to conve...
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/503#issuecomment-89710830 I added a commit to branch, but PR is not reflected. I'll try to post new PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-671) Measure tuple serialization/deserialization latency.
[ https://issues.apache.org/jira/browse/STORM-671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395832#comment-14395832 ] ASF GitHub Bot commented on STORM-671: -- Github user Parth-Brahmbhatt commented on the pull request: https://github.com/apache/storm/pull/442#issuecomment-89620855 @Lewuathe The metrics is already stored in zookeeper what you need to do is ensure the thrift structure has serialization metric as an additional field (point 2) and change the converter https://github.com/apache/storm/blob/master/storm-core/src/clj/backtype/storm/stats.clj#L369 to handle conversion from the clojure metric to thrift metrics and vice versa. Let me know if I can help in anyway. Measure tuple serialization/deserialization latency. Key: STORM-671 URL: https://issues.apache.org/jira/browse/STORM-671 Project: Apache Storm Issue Type: New Feature Reporter: Robert Joseph Evans Assignee: Kai Sasaki Some times the serialization/deserialization cost can be very high, and it is not currently measured anywhere in storm. We should measure it, at least in a similar way to how we do execute and process latency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-650) Storm-Kafka Refactoring and Improvements
[ https://issues.apache.org/jira/browse/STORM-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395839#comment-14395839 ] P. Taylor Goetz commented on STORM-650: --- [~jkreps] nm... Found the docs you referenced earlier. Storm-Kafka Refactoring and Improvements Key: STORM-650 URL: https://issues.apache.org/jira/browse/STORM-650 Project: Apache Storm Issue Type: Improvement Components: storm-kafka Reporter: P. Taylor Goetz This is intended to be a parent/umbrella JIRA covering a number of efforts/suggestions aimed at improving the storm-kafka module. The goal is to facilitate communication and collaboration by providing a central point for discussion and coordination. The first phase should be to identify and agree upon a list of high-level points we would like to address. Once that is complete, we can move on to implementation/design discussions, followed by an implementation plan, division of labor, etc. A non-exhaustive, initial list of items follows. New/additional thoughts can be proposed in the comments. * Improve API for Specifying the Kafka Starting point Configuring the kafka spout's starting position (e.g. forceFromStart=true) is a common source of confusion. This should be refactored to provide an easy to understand, unambiguous API for configuring this property. * Use Kafka APIs Instead of Internal ZK Metadata (STORM-590) Currently the Kafka spout relies on reading Kafka's internal metadata from zookeeper. This should be refactored to use the Kafka Consumer API to protect against changes to the internal metadata format stored in ZK. * Improve Error Handling There are a number of failure scenarios with the kafka spout that users may want to react to differently based on their use case. Add a failure handler API that allows users to implement and/or plug in alternative failure handling implementations. It is assumed that default (sane) implementations would be included and configured by default. * Configuration/General Refactoring (BrokerHosts, etc.) (STORM-631) (need to flesh this out better) Reduce unnecessary marker interfaces/instance of checks. Unify configuration of core storm/trident spout implementations. * Kafka Spout doesn't pick up from the beginning of the queue unless forceFromStart specified (STORM-563) Discussion Items: * How important is backward compatibility? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-650) Storm-Kafka Refactoring and Improvements
[ https://issues.apache.org/jira/browse/STORM-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395841#comment-14395841 ] Jay Kreps commented on STORM-650: - [~ptgoetz] Awesome, much appreciated. Storm-Kafka Refactoring and Improvements Key: STORM-650 URL: https://issues.apache.org/jira/browse/STORM-650 Project: Apache Storm Issue Type: Improvement Components: storm-kafka Reporter: P. Taylor Goetz This is intended to be a parent/umbrella JIRA covering a number of efforts/suggestions aimed at improving the storm-kafka module. The goal is to facilitate communication and collaboration by providing a central point for discussion and coordination. The first phase should be to identify and agree upon a list of high-level points we would like to address. Once that is complete, we can move on to implementation/design discussions, followed by an implementation plan, division of labor, etc. A non-exhaustive, initial list of items follows. New/additional thoughts can be proposed in the comments. * Improve API for Specifying the Kafka Starting point Configuring the kafka spout's starting position (e.g. forceFromStart=true) is a common source of confusion. This should be refactored to provide an easy to understand, unambiguous API for configuring this property. * Use Kafka APIs Instead of Internal ZK Metadata (STORM-590) Currently the Kafka spout relies on reading Kafka's internal metadata from zookeeper. This should be refactored to use the Kafka Consumer API to protect against changes to the internal metadata format stored in ZK. * Improve Error Handling There are a number of failure scenarios with the kafka spout that users may want to react to differently based on their use case. Add a failure handler API that allows users to implement and/or plug in alternative failure handling implementations. It is assumed that default (sane) implementations would be included and configured by default. * Configuration/General Refactoring (BrokerHosts, etc.) (STORM-631) (need to flesh this out better) Reduce unnecessary marker interfaces/instance of checks. Unify configuration of core storm/trident spout implementations. * Kafka Spout doesn't pick up from the beginning of the queue unless forceFromStart specified (STORM-563) Discussion Items: * How important is backward compatibility? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-650) Storm-Kafka Refactoring and Improvements
[ https://issues.apache.org/jira/browse/STORM-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395837#comment-14395837 ] P. Taylor Goetz commented on STORM-650: --- Thanks [~jkreps], I think that sounds like a great idea. Can you point to some docs for the new consumer API? I'll try to review the API and our code to see what, if anything, we would need. Storm-Kafka Refactoring and Improvements Key: STORM-650 URL: https://issues.apache.org/jira/browse/STORM-650 Project: Apache Storm Issue Type: Improvement Components: storm-kafka Reporter: P. Taylor Goetz This is intended to be a parent/umbrella JIRA covering a number of efforts/suggestions aimed at improving the storm-kafka module. The goal is to facilitate communication and collaboration by providing a central point for discussion and coordination. The first phase should be to identify and agree upon a list of high-level points we would like to address. Once that is complete, we can move on to implementation/design discussions, followed by an implementation plan, division of labor, etc. A non-exhaustive, initial list of items follows. New/additional thoughts can be proposed in the comments. * Improve API for Specifying the Kafka Starting point Configuring the kafka spout's starting position (e.g. forceFromStart=true) is a common source of confusion. This should be refactored to provide an easy to understand, unambiguous API for configuring this property. * Use Kafka APIs Instead of Internal ZK Metadata (STORM-590) Currently the Kafka spout relies on reading Kafka's internal metadata from zookeeper. This should be refactored to use the Kafka Consumer API to protect against changes to the internal metadata format stored in ZK. * Improve Error Handling There are a number of failure scenarios with the kafka spout that users may want to react to differently based on their use case. Add a failure handler API that allows users to implement and/or plug in alternative failure handling implementations. It is assumed that default (sane) implementations would be included and configured by default. * Configuration/General Refactoring (BrokerHosts, etc.) (STORM-631) (need to flesh this out better) Reduce unnecessary marker interfaces/instance of checks. Unify configuration of core storm/trident spout implementations. * Kafka Spout doesn't pick up from the beginning of the queue unless forceFromStart specified (STORM-563) Discussion Items: * How important is backward compatibility? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-745) Second Commandline Parameter passed to the main class is skipped when run in windows
[
https://issues.apache.org/jira/browse/STORM-745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395971#comment-14395971
]
Jungtaek Lim commented on STORM-745:
[~jishnu111]
Hello.
I'd like to ask some question to get into it.
1. Did you download Storm from http://storm.apache.org/downloads.html ? What
does pre-built for windows mean?
2. What file did you use for run? Is it storm.py or storm.cmd?
Second Commandline Parameter passed to the main class is skipped when run in
windows
Key: STORM-745
URL: https://issues.apache.org/jira/browse/STORM-745
Project: Apache Storm
Issue Type: Bug
Affects Versions: 0.9.3-rc2, 0.9.4
Environment: Windows Only
Reporter: jishnu
Labels: 2nd, command-line, parameter, skipping, windows
Always the second parameter is getting skipped.
E:\targetstorm jar StormZeroMQ.jar com.wipro.bdas.zeromq.ZMQTopology value1
value2 value3 value4 value5
Output
I=0 value=value1
I=1 value=value3
I=2 value=value4
I=3 value=value5
public class ZMQTopology {
public static void main(String[] args) throws AlreadyAliveException,
InvalidTopologyException {
for(int i=0;i(args.length);i++)
{ System.out.println(I= +i+
value=+args[i]);
}
I am using the apache storm pre-built for windows.
After some amount of debugging I could find that it happens only with windows
machine . I was able to reproduce the error in 2 windows machine. With both
0.9.3 and 0.9.4 .In Linux machine I could see command line parameters working
perfectly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-748) Package Multi-Lang scripts so they don't have to be duplicated
[ https://issues.apache.org/jira/browse/STORM-748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395983#comment-14395983 ] Jungtaek Lim commented on STORM-748: Strongly agreed. I'd like to know how we can package it. Do you mean building python module / ruby gem / nodejs module? If then maybe issue is 'how to use module before releasing'. Package Multi-Lang scripts so they don't have to be duplicated -- Key: STORM-748 URL: https://issues.apache.org/jira/browse/STORM-748 Project: Apache Storm Issue Type: Improvement Reporter: P. Taylor Goetz Assignee: P. Taylor Goetz We currently have multiple copies of storm's multilang scripts throughout the codebase, which makes changing them tedious and error prone. They should be packaged in such a way as to allow them to be defined in one place (without symlinks) and re-used without duplication. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-752) [storm-redis] Clarify Redis*StateUpdater's expire is optional
[ https://issues.apache.org/jira/browse/STORM-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395988#comment-14395988 ] ASF GitHub Bot commented on STORM-752: -- Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/488#issuecomment-89679084 Filed a issue to JIRA. [storm-redis] Clarify Redis*StateUpdater's expire is optional - Key: STORM-752 URL: https://issues.apache.org/jira/browse/STORM-752 Project: Apache Storm Issue Type: Improvement Affects Versions: 0.10.0, 0.11.0 Reporter: Jungtaek Lim Assignee: Jungtaek Lim It would be better to clarify Redis*StateUpdater's expire is optional. You can find relevant conversation on link. https://issues.apache.org/jira/browse/STORM-723?focusedCommentId=14381221page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14381221 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-752] [storm-redis] Clarify Redis*StateU...
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/488#issuecomment-89679084 Filed a issue to JIRA. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (STORM-752) [storm-redis] Clarify Redis*StateUpdater's expire is optional
Jungtaek Lim created STORM-752: -- Summary: [storm-redis] Clarify Redis*StateUpdater's expire is optional Key: STORM-752 URL: https://issues.apache.org/jira/browse/STORM-752 Project: Apache Storm Issue Type: Improvement Affects Versions: 0.10.0, 0.11.0 Reporter: Jungtaek Lim Assignee: Jungtaek Lim It would be better to clarify Redis*StateUpdater's expire is optional. You can find relevant conversation on link. https://issues.apache.org/jira/browse/STORM-723?focusedCommentId=14381221page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14381221 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-753) [storm-redis] Let Redis*StateUpdater to have additional mapper - converting tuple to key/value
[ https://issues.apache.org/jira/browse/STORM-753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-753: --- Description: https://issues.apache.org/jira/browse/STORM-723?focusedCommentId=14381626page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14381626 Currently Redis*StateUpdater use key prefix to let users customize key name. Since it's less powerful, how about having 2 mappers which one is for converting tuple to key/value, and another one is for converting value from Redis to Storm Values? You can refer https://github.com/apache/storm/blob/master/external/storm-hbase/src/main/java/org/apache/storm/hbase/trident/state/HBaseState.java. was: Currently Redis*StateUpdater use key prefix to let users customize key name. Since it's less powerful, how about having 2 mappers which one is for converting tuple to key/value, and another one is for converting value from Redis to Storm Values? You can refer https://github.com/apache/storm/blob/master/external/storm-hbase/src/main/java/org/apache/storm/hbase/trident/state/HBaseState.java. [storm-redis] Let Redis*StateUpdater to have additional mapper - converting tuple to key/value -- Key: STORM-753 URL: https://issues.apache.org/jira/browse/STORM-753 Project: Apache Storm Issue Type: Improvement Affects Versions: 0.10.0, 0.11.0 Reporter: Jungtaek Lim Assignee: Jungtaek Lim https://issues.apache.org/jira/browse/STORM-723?focusedCommentId=14381626page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14381626 Currently Redis*StateUpdater use key prefix to let users customize key name. Since it's less powerful, how about having 2 mappers which one is for converting tuple to key/value, and another one is for converting value from Redis to Storm Values? You can refer https://github.com/apache/storm/blob/master/external/storm-hbase/src/main/java/org/apache/storm/hbase/trident/state/HBaseState.java. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-753) [storm-redis] Let Redis*StateUpdater to have additional mapper - converting Redis key/value to Storm Values
[ https://issues.apache.org/jira/browse/STORM-753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-753: --- Summary: [storm-redis] Let Redis*StateUpdater to have additional mapper - converting Redis key/value to Storm Values (was: [storm-redis] Let Redis*StateUpdater to have additional mapper - converting tuple to key/value) [storm-redis] Let Redis*StateUpdater to have additional mapper - converting Redis key/value to Storm Values --- Key: STORM-753 URL: https://issues.apache.org/jira/browse/STORM-753 Project: Apache Storm Issue Type: Improvement Affects Versions: 0.10.0, 0.11.0 Reporter: Jungtaek Lim Assignee: Jungtaek Lim https://issues.apache.org/jira/browse/STORM-723?focusedCommentId=14381626page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14381626 Currently Redis*StateUpdater use key prefix to let users customize key name. Since it's less powerful, how about having 2 mappers which one is for converting tuple to key/value, and another one is for converting value from Redis to Storm Values? You can refer https://github.com/apache/storm/blob/master/external/storm-hbase/src/main/java/org/apache/storm/hbase/trident/state/HBaseState.java. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
