[jira] [Updated] (STORM-1231) port backtype.storm.scheduler.EvenScheduler to java
[ https://issues.apache.org/jira/browse/STORM-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Wang updated STORM-1231: Attachment: (was: storm-scheduler-issue-fixing.png) > port backtype.storm.scheduler.EvenScheduler to java > --- > > Key: STORM-1231 > URL: https://issues.apache.org/jira/browse/STORM-1231 > Project: Apache Storm > Issue Type: New Feature >Reporter: Robert Joseph Evans >Assignee: Xin Wang > Labels: java-migration, jstorm-merger > > Port the even scheduler to java. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (STORM-1231) port backtype.storm.scheduler.EvenScheduler to java
[ https://issues.apache.org/jira/browse/STORM-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Wang updated STORM-1231: Comment: was deleted (was: PR(https://github.com/apache/storm/pull/1108) ported _backtype.storm.scheduler.EvenScheduler_ to java, and fixed the load balancing issue. load balancing issue: We have a 3 nodes(supervisor1,supervisor2,supervisor3) cluster, each node has 3 available slots. And we submit two topolopies, the first has 2 workers and the second has 4 workers. total available slots:[supervisor1:6700, supervisor2:6700, supervisor3:6700, supervisor1:6701, supervisor2:6701, supervisor3:6701, supervisor1:6702, supervisor2:6702, supervisor3:6702] *before fix* first assign:[supervisor1:6700, supervisor2:6700] current available slots:[supervisor1:6701, supervisor2:6701, supervisor3:6700, supervisor1:6702, supervisor2:6702, supervisor3:6701, supervisor3:6702] second assign:[supervisor1:6701, supervisor2:6701, supervisor3:6700, supervisor1:6702] *after fix:* first assign:[supervisor1:6700, supervisor2:6700] current available slots:[supervisor3:6700, supervisor1:6701, supervisor2:6701, supervisor3:6701, supervisor1:6702, supervisor2:6702, supervisor3:6702] second assign:[supervisor3:6700, supervisor1:6701, supervisor2:6701, supervisor3:6701] !https://issues.apache.org/jira/secure/attachment/12788021/storm-scheduler-issue-fixing.png!) > port backtype.storm.scheduler.EvenScheduler to java > --- > > Key: STORM-1231 > URL: https://issues.apache.org/jira/browse/STORM-1231 > Project: Apache Storm > Issue Type: New Feature >Reporter: Robert Joseph Evans >Assignee: Xin Wang > Labels: java-migration, jstorm-merger > > Port the even scheduler to java. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (STORM-1231) port backtype.storm.scheduler.EvenScheduler to java
[ https://issues.apache.org/jira/browse/STORM-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Wang updated STORM-1231: Comment: was deleted (was: I attached an image file to make it easier to understand.) > port backtype.storm.scheduler.EvenScheduler to java > --- > > Key: STORM-1231 > URL: https://issues.apache.org/jira/browse/STORM-1231 > Project: Apache Storm > Issue Type: New Feature >Reporter: Robert Joseph Evans >Assignee: Xin Wang > Labels: java-migration, jstorm-merger > > Port the even scheduler to java. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1232] port backtype.storm.scheduler.Def...
Github user vesense commented on a diff in the pull request:
https://github.com/apache/storm/pull/1108#discussion_r53563947
--- Diff: storm-core/src/jvm/org/apache/storm/scheduler/EvenScheduler.java
---
@@ -0,0 +1,157 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.scheduler;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.TreeMap;
+
+import org.apache.storm.utils.Utils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+
+public class EvenScheduler implements IScheduler {
+private static final Logger LOG =
LoggerFactory.getLogger(EvenScheduler.class);
+
+public static List sortSlots(List
availableSlots, Cluster cluster) {
--- End diff --
Done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-1232) port backtype.storm.scheduler.DefaultScheduler to java
[
https://issues.apache.org/jira/browse/STORM-1232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15155966#comment-15155966
]
ASF GitHub Bot commented on STORM-1232:
---
Github user vesense commented on a diff in the pull request:
https://github.com/apache/storm/pull/1108#discussion_r53563947
--- Diff: storm-core/src/jvm/org/apache/storm/scheduler/EvenScheduler.java
---
@@ -0,0 +1,157 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.scheduler;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.TreeMap;
+
+import org.apache.storm.utils.Utils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+
+public class EvenScheduler implements IScheduler {
+private static final Logger LOG =
LoggerFactory.getLogger(EvenScheduler.class);
+
+public static List sortSlots(List
availableSlots, Cluster cluster) {
--- End diff --
Done.
> port backtype.storm.scheduler.DefaultScheduler to java
> ---
>
> Key: STORM-1232
> URL: https://issues.apache.org/jira/browse/STORM-1232
> Project: Apache Storm
> Issue Type: New Feature
> Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Xin Wang
> Labels: java-migration, jstorm-merger
>
> port the DefaultScheduler to java
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: [STORM-1232] port backtype.storm.scheduler.Def...
Github user vesense commented on the pull request: https://github.com/apache/storm/pull/1108#issuecomment-186782200 @abhishekagarwal87 The build failure has been fixed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1232) port backtype.storm.scheduler.DefaultScheduler to java
[ https://issues.apache.org/jira/browse/STORM-1232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15155969#comment-15155969 ] ASF GitHub Bot commented on STORM-1232: --- Github user vesense commented on the pull request: https://github.com/apache/storm/pull/1108#issuecomment-186782200 @abhishekagarwal87 The build failure has been fixed. > port backtype.storm.scheduler.DefaultScheduler to java > --- > > Key: STORM-1232 > URL: https://issues.apache.org/jira/browse/STORM-1232 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: Xin Wang > Labels: java-migration, jstorm-merger > > port the DefaultScheduler to java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1232] port backtype.storm.scheduler.Def...
Github user vesense commented on the pull request: https://github.com/apache/storm/pull/1108#issuecomment-186782799 @ptgoetz @revans2 @jerrypeng Could you also take a look? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1232) port backtype.storm.scheduler.DefaultScheduler to java
[ https://issues.apache.org/jira/browse/STORM-1232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15155971#comment-15155971 ] ASF GitHub Bot commented on STORM-1232: --- Github user vesense commented on the pull request: https://github.com/apache/storm/pull/1108#issuecomment-186782799 @ptgoetz @revans2 @jerrypeng Could you also take a look? > port backtype.storm.scheduler.DefaultScheduler to java > --- > > Key: STORM-1232 > URL: https://issues.apache.org/jira/browse/STORM-1232 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: Xin Wang > Labels: java-migration, jstorm-merger > > port the DefaultScheduler to java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-1565) Multi-Lang Performance Improvements
Xin Wang created STORM-1565:
---
Summary: Multi-Lang Performance Improvements
Key: STORM-1565
URL: https://issues.apache.org/jira/browse/STORM-1565
Project: Apache Storm
Issue Type: Improvement
Components: storm-core
Reporter: Xin Wang
Assignee: Xin Wang
1. add _org.apache.storm.multilang.MessagePackSerializer_
2. change default "topology.multilang.serializer" to _MessagePackSerializer_
According to http://msgpack.org/: It's like JSON, but fast and small.
{quote}
MessagePack is an efficient binary serialization format. It lets you exchange
data among multiple languages like JSON. But it's faster and smaller. Small
integers are encoded into a single byte, and typical short strings require only
one extra byte in addition to the strings themselves.
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (STORM-1565) Multi-Lang Performance Improvements
[
https://issues.apache.org/jira/browse/STORM-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xin Wang updated STORM-1565:
Description:
1. add _org.apache.storm.multilang.MessagePackSerializer_
2. change default "topology.multilang.serializer" to _MessagePackSerializer_
According to http://msgpack.org/ : It's like JSON, but fast and small.
{quote}
MessagePack is an efficient binary serialization format. It lets you exchange
data among multiple languages like JSON. But it's faster and smaller. Small
integers are encoded into a single byte, and typical short strings require only
one extra byte in addition to the strings themselves.
{quote}
was:
1. add _org.apache.storm.multilang.MessagePackSerializer_
2. change default "topology.multilang.serializer" to _MessagePackSerializer_
According to http://msgpack.org/: It's like JSON, but fast and small.
{quote}
MessagePack is an efficient binary serialization format. It lets you exchange
data among multiple languages like JSON. But it's faster and smaller. Small
integers are encoded into a single byte, and typical short strings require only
one extra byte in addition to the strings themselves.
{quote}
> Multi-Lang Performance Improvements
> ---
>
> Key: STORM-1565
> URL: https://issues.apache.org/jira/browse/STORM-1565
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-core
>Reporter: Xin Wang
>Assignee: Xin Wang
>
> 1. add _org.apache.storm.multilang.MessagePackSerializer_
> 2. change default "topology.multilang.serializer" to _MessagePackSerializer_
> According to http://msgpack.org/ : It's like JSON, but fast and small.
> {quote}
> MessagePack is an efficient binary serialization format. It lets you exchange
> data among multiple languages like JSON. But it's faster and smaller. Small
> integers are encoded into a single byte, and typical short strings require
> only one extra byte in addition to the strings themselves.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user jianbzhou commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-186820101 Hi **@hmcl **, is there any update on the patch please? thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-822) As a storm developer I’d like to use the new kafka consumer API (0.8.3) to reduce dependencies and use long term supported kafka apis
[ https://issues.apache.org/jira/browse/STORM-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15156009#comment-15156009 ] ASF GitHub Bot commented on STORM-822: -- Github user jianbzhou commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-186820101 Hi **@hmcl **, is there any update on the patch please? thanks. > As a storm developer I’d like to use the new kafka consumer API (0.8.3) to > reduce dependencies and use long term supported kafka apis > -- > > Key: STORM-822 > URL: https://issues.apache.org/jira/browse/STORM-822 > Project: Apache Storm > Issue Type: Story > Components: storm-kafka >Reporter: Thomas Becker >Assignee: Hugo Louro > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1523] util.clj available-port conversio...
Github user abhishekagarwal87 commented on the pull request: https://github.com/apache/storm/pull/1073#issuecomment-186830938 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1523) Translate Available Ports task in util.java
[ https://issues.apache.org/jira/browse/STORM-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15156036#comment-15156036 ] ASF GitHub Bot commented on STORM-1523: --- Github user abhishekagarwal87 commented on the pull request: https://github.com/apache/storm/pull/1073#issuecomment-186830938 +1 > Translate Available Ports task in util.java > --- > > Key: STORM-1523 > URL: https://issues.apache.org/jira/browse/STORM-1523 > Project: Apache Storm > Issue Type: Sub-task > Components: storm-core >Reporter: Sanket Reddy > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-917 : Ability to emit from the spout to ...
Github user abhishekagarwal87 commented on the pull request: https://github.com/apache/storm/pull/601#issuecomment-186855234 Thanks for doing this Anishek. My comments Kafka spout should declare the stream id in output declaration. ``` declarer.declare(streamID, _spoutConfig.scheme.getOutputFields() ``` I would prefer the streamID to come from the scheme. More than often the stream on which data is to be sent to, depends on the input data. Since scheme knows about the data, it can also determine the stream. This makes more sense when you think of multiple output streams from same spout. The only downside I see is that the scheme interface will change and that might translate into compatibility issues. Nonetheless this PR is a good start. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-917 : Ability to emit from the spout to ...
Github user abhishekagarwal87 commented on a diff in the pull request:
https://github.com/apache/storm/pull/601#discussion_r53569055
--- Diff: external/storm-kafka/src/jvm/storm/kafka/PartitionManager.java ---
@@ -126,7 +127,7 @@ public Map getMetricsDataMap() {
}
//returns false if it's reached the end of current batch
-public EmitState next(SpoutOutputCollector collector) {
+public EmitState next(SpoutOutputCollector collector, String
emitStreamId) {
--- End diff --
what is the need of additional argument since spoutConfig is available to
this class. stream name can be initialized once in the prepare phase itself.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-917) Ability to emit messages from a kafka spout to a specific stream
[ https://issues.apache.org/jira/browse/STORM-917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15156076#comment-15156076 ] ASF GitHub Bot commented on STORM-917: -- Github user abhishekagarwal87 commented on the pull request: https://github.com/apache/storm/pull/601#issuecomment-186855234 Thanks for doing this Anishek. My comments Kafka spout should declare the stream id in output declaration. ``` declarer.declare(streamID, _spoutConfig.scheme.getOutputFields() ``` I would prefer the streamID to come from the scheme. More than often the stream on which data is to be sent to, depends on the input data. Since scheme knows about the data, it can also determine the stream. This makes more sense when you think of multiple output streams from same spout. The only downside I see is that the scheme interface will change and that might translate into compatibility issues. Nonetheless this PR is a good start. > Ability to emit messages from a kafka spout to a specific stream > - > > Key: STORM-917 > URL: https://issues.apache.org/jira/browse/STORM-917 > Project: Apache Storm > Issue Type: Improvement > Components: storm-kafka >Reporter: anishek > Fix For: 1.0.0 > > > Today even if we have multiple spouts in storm they will all emit on the same > "default" stream. Having an ability to configure the stream that a kafka > spout emits to would be great. By default should emit to the "default" stream -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-917) Ability to emit messages from a kafka spout to a specific stream
[
https://issues.apache.org/jira/browse/STORM-917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15156077#comment-15156077
]
ASF GitHub Bot commented on STORM-917:
--
Github user abhishekagarwal87 commented on a diff in the pull request:
https://github.com/apache/storm/pull/601#discussion_r53569055
--- Diff: external/storm-kafka/src/jvm/storm/kafka/PartitionManager.java ---
@@ -126,7 +127,7 @@ public Map getMetricsDataMap() {
}
//returns false if it's reached the end of current batch
-public EmitState next(SpoutOutputCollector collector) {
+public EmitState next(SpoutOutputCollector collector, String
emitStreamId) {
--- End diff --
what is the need of additional argument since spoutConfig is available to
this class. stream name can be initialized once in the prepare phase itself.
> Ability to emit messages from a kafka spout to a specific stream
> -
>
> Key: STORM-917
> URL: https://issues.apache.org/jira/browse/STORM-917
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-kafka
>Reporter: anishek
> Fix For: 1.0.0
>
>
> Today even if we have multiple spouts in storm they will all emit on the same
> "default" stream. Having an ability to configure the stream that a kafka
> spout emits to would be great. By default should emit to the "default" stream
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-917 : Ability to emit from the spout to ...
Github user abhishekagarwal87 commented on a diff in the pull request: https://github.com/apache/storm/pull/601#discussion_r53569108 --- Diff: external/storm-kafka/src/jvm/storm/kafka/SpoutConfig.java --- @@ -17,6 +17,8 @@ */ package storm.kafka; +import backtype.storm.utils.Utils; --- End diff -- You need to upmerge your master branch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-917) Ability to emit messages from a kafka spout to a specific stream
[ https://issues.apache.org/jira/browse/STORM-917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15156080#comment-15156080 ] ASF GitHub Bot commented on STORM-917: -- Github user abhishekagarwal87 commented on a diff in the pull request: https://github.com/apache/storm/pull/601#discussion_r53569108 --- Diff: external/storm-kafka/src/jvm/storm/kafka/SpoutConfig.java --- @@ -17,6 +17,8 @@ */ package storm.kafka; +import backtype.storm.utils.Utils; --- End diff -- You need to upmerge your master branch. > Ability to emit messages from a kafka spout to a specific stream > - > > Key: STORM-917 > URL: https://issues.apache.org/jira/browse/STORM-917 > Project: Apache Storm > Issue Type: Improvement > Components: storm-kafka >Reporter: anishek > Fix For: 1.0.0 > > > Today even if we have multiple spouts in storm they will all emit on the same > "default" stream. Having an ability to configure the stream that a kafka > spout emits to would be great. By default should emit to the "default" stream -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1552] Fix topology event sampling log d...
Github user arunmahadevan commented on the pull request: https://github.com/apache/storm/pull/1127#issuecomment-186883739 @harshach `storm.home` is not set during unit tests so the logs were created under wrong directory. Commit https://github.com/apache/storm/commit/f3baed0d4bb9c415657dfa97f33519994e991e17 fixes it. Please take a look. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1552) Fix topology event sampling log directory
[
https://issues.apache.org/jira/browse/STORM-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15156155#comment-15156155
]
ASF GitHub Bot commented on STORM-1552:
---
Github user arunmahadevan commented on the pull request:
https://github.com/apache/storm/pull/1127#issuecomment-186883739
@harshach `storm.home` is not set during unit tests so the logs were
created under wrong directory. Commit
https://github.com/apache/storm/commit/f3baed0d4bb9c415657dfa97f33519994e991e17
fixes it. Please take a look.
> Fix topology event sampling log directory
> --
>
> Key: STORM-1552
> URL: https://issues.apache.org/jira/browse/STORM-1552
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Arun Mahadevan
>Assignee: Arun Mahadevan
>
> Run a topology and enable event inspection by clicking "Debug" from UI. The
> events are logged under
> "storm-local/workers-artifacts/{storm-id}/port/events.log". In the spout/bolt
> details page, the "events" link does not display the log file.
> The events.log should be kept under
> logs/workers-artifacts/{storm-id}/{port}/events.log so that its viewable via
> logviewer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: closes STORM-1545 and STORM-1552
Github user mproch commented on the pull request: https://github.com/apache/storm/pull/1123#issuecomment-186895540 I removed relative path handling and updated the title - hope this is what you meant? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1545) Topology Debug Event Log in Wrong Location
[
https://issues.apache.org/jira/browse/STORM-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15156183#comment-15156183
]
ASF GitHub Bot commented on STORM-1545:
---
Github user mproch commented on the pull request:
https://github.com/apache/storm/pull/1123#issuecomment-186895540
I removed relative path handling and updated the title - hope this is what
you meant?
> Topology Debug Event Log in Wrong Location
> --
>
> Key: STORM-1545
> URL: https://issues.apache.org/jira/browse/STORM-1545
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core, storm-ui
>Affects Versions: 1.0.0
>Reporter: P. Taylor Goetz
> Labels: newbie
>
> Currently the {{events.log}} file is not created where the log viewer expects
> it so be, so the "events" link under debug in Storm UI returns a 404.
> The file should be in:
> {{$storm.log.dir/workers-artifacts/$topology_id/$port/events.log}}
> but is instead in:
> {{$storm.local.dir/workers-artifacts/$topology_id/$port/events.log}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-822: Kafka Spout New Consumer API
GitHub user hmcl opened a pull request: https://github.com/apache/storm/pull/1131 STORM-822: Kafka Spout New Consumer API This patch is still under development and was uploaded at this moment for early testing. Please read README. There may be a bug in the offsets management, because of diff o 1. I am looking into it. Currently polling from an arbitrary offset is possible but it will come in the next patch, today or tomorrow I refactored the code a bit and left, maybe, some unnecessary locking. I am also looking into it. @connieyang @jianbzhou @tgravescs please let me know of any other requirements you may have and I will address them soon. Thanks You can merge this pull request into a Git repository by running: $ git pull https://github.com/hmcl/storm-apache STORM-822 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1131.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1131 commit bcabde29a0df7713b71d1807afec3be5b0147664 Author: Hugo Louro Date: 2015-12-14T18:16:42Z STORM-822 - Storm Kafka Spout new consumer API commit 92ae8a6aff4a1bd66c9153dcfd2c44d16de37637 Author: Hugo Louro Date: 2016-02-21T22:11:12Z STORM-822 - Storm Kafka Spout new consumer API --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user hmcl commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-186931043 @connieyang @jianbzhou @tgravescs patch is here: https://github.com/apache/storm/pull/1131 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-822) As a storm developer I’d like to use the new kafka consumer API (0.8.3) to reduce dependencies and use long term supported kafka apis
[ https://issues.apache.org/jira/browse/STORM-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15156254#comment-15156254 ] ASF GitHub Bot commented on STORM-822: -- GitHub user hmcl opened a pull request: https://github.com/apache/storm/pull/1131 STORM-822: Kafka Spout New Consumer API This patch is still under development and was uploaded at this moment for early testing. Please read README. There may be a bug in the offsets management, because of diff o 1. I am looking into it. Currently polling from an arbitrary offset is possible but it will come in the next patch, today or tomorrow I refactored the code a bit and left, maybe, some unnecessary locking. I am also looking into it. @connieyang @jianbzhou @tgravescs please let me know of any other requirements you may have and I will address them soon. Thanks You can merge this pull request into a Git repository by running: $ git pull https://github.com/hmcl/storm-apache STORM-822 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1131.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1131 commit bcabde29a0df7713b71d1807afec3be5b0147664 Author: Hugo Louro Date: 2015-12-14T18:16:42Z STORM-822 - Storm Kafka Spout new consumer API commit 92ae8a6aff4a1bd66c9153dcfd2c44d16de37637 Author: Hugo Louro Date: 2016-02-21T22:11:12Z STORM-822 - Storm Kafka Spout new consumer API > As a storm developer I’d like to use the new kafka consumer API (0.8.3) to > reduce dependencies and use long term supported kafka apis > -- > > Key: STORM-822 > URL: https://issues.apache.org/jira/browse/STORM-822 > Project: Apache Storm > Issue Type: Story > Components: storm-kafka >Reporter: Thomas Becker >Assignee: Hugo Louro > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-822) As a storm developer I’d like to use the new kafka consumer API (0.8.3) to reduce dependencies and use long term supported kafka apis
[ https://issues.apache.org/jira/browse/STORM-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15156255#comment-15156255 ] ASF GitHub Bot commented on STORM-822: -- Github user hmcl commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-186931043 @connieyang @jianbzhou @tgravescs patch is here: https://github.com/apache/storm/pull/1131 > As a storm developer I’d like to use the new kafka consumer API (0.8.3) to > reduce dependencies and use long term supported kafka apis > -- > > Key: STORM-822 > URL: https://issues.apache.org/jira/browse/STORM-822 > Project: Apache Storm > Issue Type: Story > Components: storm-kafka >Reporter: Thomas Becker >Assignee: Hugo Louro > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (STORM-1286) port backtype.storm.command.kill-workers to java
[ https://issues.apache.org/jira/browse/STORM-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fang reassigned STORM-1286: Assignee: John Fang > port backtype.storm.command.kill-workers to java > > > Key: STORM-1286 > URL: https://issues.apache.org/jira/browse/STORM-1286 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: John Fang > Labels: java-migration, jstorm-merger > > Kill all running worker processes on this node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1065) storm-kafka : kafka-partition can not find leader in zookeeper
[ https://issues.apache.org/jira/browse/STORM-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15156423#comment-15156423 ] ASF GitHub Bot commented on STORM-1065: --- GitHub user lujinhong opened a pull request: https://github.com/apache/storm/pull/1132 storm-kafka : kafka-partition can not find leader in zookeeper #1132 STORM-1065. storm-kafka : kafka-partition can not find leader in zookeeper You can merge this pull request into a Git repository by running: $ git pull https://github.com/lujinhong/storm master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1132.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1132 commit 960616274ca655408a143f984e48199efcc5fe7f Author: lujinhong Date: 2016-02-22T03:00:07Z fix STORM-1065 STORM-1065. storm-kafka : kafka-partition can not find leader in zookeeper > storm-kafka : kafka-partition can not find leader in zookeeper > --- > > Key: STORM-1065 > URL: https://issues.apache.org/jira/browse/STORM-1065 > Project: Apache Storm > Issue Type: Improvement > Components: storm-kafka >Reporter: dongxinwang >Priority: Minor > > If the Kafka cluster is not consistent with the zookeeper data, the partition > can not find leader in zookeeper.In storm-kafka, it throws runtime > exception"No leader found for partition",it is not friendly. > Suggestion: > If there is no leader partition in zookeeper,don't add the partition to > GlobalPartitionInformation object,instead of throwing runtime exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: storm-kafka : kafka-partition can not find lea...
GitHub user lujinhong opened a pull request: https://github.com/apache/storm/pull/1132 storm-kafka : kafka-partition can not find leader in zookeeper #1132 STORM-1065. storm-kafka : kafka-partition can not find leader in zookeeper You can merge this pull request into a Git repository by running: $ git pull https://github.com/lujinhong/storm master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1132.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1132 commit 960616274ca655408a143f984e48199efcc5fe7f Author: lujinhong Date: 2016-02-22T03:00:07Z fix STORM-1065 STORM-1065. storm-kafka : kafka-partition can not find leader in zookeeper --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822: Kafka Spout New Consumer API
Github user lujinhong commented on the pull request: https://github.com/apache/storm/pull/1131#issuecomment-186986537 In KafkaUtil.java, NOT_LEADER_FOR_PARTITION happend, may catch the NOT_LEADER_FOR_PARTITION execption will be a good idea. This exception should be catch and then log & ignored Or invole the ZkCoordinator#refresh(). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-822) As a storm developer I’d like to use the new kafka consumer API (0.8.3) to reduce dependencies and use long term supported kafka apis
[ https://issues.apache.org/jira/browse/STORM-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15156440#comment-15156440 ] ASF GitHub Bot commented on STORM-822: -- Github user lujinhong commented on the pull request: https://github.com/apache/storm/pull/1131#issuecomment-186986537 In KafkaUtil.java, NOT_LEADER_FOR_PARTITION happend, may catch the NOT_LEADER_FOR_PARTITION execption will be a good idea. This exception should be catch and then log & ignored Or invole the ZkCoordinator#refresh(). > As a storm developer I’d like to use the new kafka consumer API (0.8.3) to > reduce dependencies and use long term supported kafka apis > -- > > Key: STORM-822 > URL: https://issues.apache.org/jira/browse/STORM-822 > Project: Apache Storm > Issue Type: Story > Components: storm-kafka >Reporter: Thomas Becker >Assignee: Hugo Louro > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: storm-kafka : kafka-partition can not find lea...
Github user lujinhong commented on the pull request: https://github.com/apache/storm/pull/1132#issuecomment-186987218 In KafkaUtil.java, NOT_LEADER_FOR_PARTITION happend, catching the NOT_LEADER_FOR_PARTITION execption will be a good idea. This exception should be catch and then log & ignored Or invole the ZkCoordinator#refresh(). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: closes STORM-1545 and STORM-1552
Github user arunmahadevan commented on the pull request: https://github.com/apache/storm/pull/1123#issuecomment-187022132 When the unit tests are run it creates log files under /logs. It should ideally be under system tmp directory. Maybe unrelated, but when an example topology is submitted worker crashes with the below error, `2016-02-22 10:56:51.085 o.a.s.d.worker [ERROR] Error on initialization of server mk-worker java.lang.ClassCastException: java.lang.String cannot be cast to java.io.File at org.apache.storm.daemon.worker$fn__6265$exec_fn__1730__auto6266.invoke(worker.clj:622) ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] at clojure.lang.AFn.applyToHelper(AFn.java:178) ~[clojure-1.7.0.jar:?] at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?] at clojure.core$apply.invoke(core.clj:630) ~[clojure-1.7.0.jar:?] at org.apache.storm.daemon.worker$fn__6265$mk_worker__6362.doInvoke(worker.clj:613) [storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.7.0.jar:?] at org.apache.storm.daemon.worker$_main.invoke(worker.clj:810) [storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] at clojure.lang.AFn.applyToHelper(AFn.java:165) [clojure-1.7.0.jar:?] at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.7.0.jar:?] at org.apache.storm.daemon.worker.main(Unknown Source) [storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1545) Topology Debug Event Log in Wrong Location
[
https://issues.apache.org/jira/browse/STORM-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15156496#comment-15156496
]
ASF GitHub Bot commented on STORM-1545:
---
Github user arunmahadevan commented on the pull request:
https://github.com/apache/storm/pull/1123#issuecomment-187022132
When the unit tests are run it creates log files under /logs. It should
ideally be under system tmp directory.
Maybe unrelated, but when an example topology is submitted worker crashes
with the below error,
`2016-02-22 10:56:51.085 o.a.s.d.worker [ERROR] Error on initialization of
server mk-worker
java.lang.ClassCastException: java.lang.String cannot be cast to
java.io.File
at
org.apache.storm.daemon.worker$fn__6265$exec_fn__1730__auto6266.invoke(worker.clj:622)
~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
at clojure.lang.AFn.applyToHelper(AFn.java:178) ~[clojure-1.7.0.jar:?]
at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]
at clojure.core$apply.invoke(core.clj:630) ~[clojure-1.7.0.jar:?]
at
org.apache.storm.daemon.worker$fn__6265$mk_worker__6362.doInvoke(worker.clj:613)
[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.7.0.jar:?]
at org.apache.storm.daemon.worker$_main.invoke(worker.clj:810)
[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
at clojure.lang.AFn.applyToHelper(AFn.java:165) [clojure-1.7.0.jar:?]
at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.7.0.jar:?]
at org.apache.storm.daemon.worker.main(Unknown Source)
[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]`
> Topology Debug Event Log in Wrong Location
> --
>
> Key: STORM-1545
> URL: https://issues.apache.org/jira/browse/STORM-1545
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core, storm-ui
>Affects Versions: 1.0.0
>Reporter: P. Taylor Goetz
> Labels: newbie
>
> Currently the {{events.log}} file is not created where the log viewer expects
> it so be, so the "events" link under debug in Storm UI returns a 404.
> The file should be in:
> {{$storm.log.dir/workers-artifacts/$topology_id/$port/events.log}}
> but is instead in:
> {{$storm.local.dir/workers-artifacts/$topology_id/$port/events.log}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (STORM-1566) Worker exits with error o.a.s.d.worker [ERROR] Error on initialization of server mk-worker java.lang.ClassCastException: java.lang.String cannot be cast to java.io.File
Satish Duggana created STORM-1566: - Summary: Worker exits with error o.a.s.d.worker [ERROR] Error on initialization of server mk-worker java.lang.ClassCastException: java.lang.String cannot be cast to java.io.File Key: STORM-1566 URL: https://issues.apache.org/jira/browse/STORM-1566 Project: Apache Storm Issue Type: Bug Components: storm-core Affects Versions: 2.0.0 Reporter: Satish Duggana Assignee: Satish Duggana -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: closes STORM-1545 and STORM-1552
Github user satishd commented on the pull request: https://github.com/apache/storm/pull/1123#issuecomment-187024988 https://issues.apache.org/jira/browse/STORM-1566 is raised for the above issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1545) Topology Debug Event Log in Wrong Location
[
https://issues.apache.org/jira/browse/STORM-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15156515#comment-15156515
]
ASF GitHub Bot commented on STORM-1545:
---
Github user satishd commented on the pull request:
https://github.com/apache/storm/pull/1123#issuecomment-187024988
https://issues.apache.org/jira/browse/STORM-1566 is raised for the above
issue.
> Topology Debug Event Log in Wrong Location
> --
>
> Key: STORM-1545
> URL: https://issues.apache.org/jira/browse/STORM-1545
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core, storm-ui
>Affects Versions: 1.0.0
>Reporter: P. Taylor Goetz
> Labels: newbie
>
> Currently the {{events.log}} file is not created where the log viewer expects
> it so be, so the "events" link under debug in Storm UI returns a 404.
> The file should be in:
> {{$storm.log.dir/workers-artifacts/$topology_id/$port/events.log}}
> but is instead in:
> {{$storm.local.dir/workers-artifacts/$topology_id/$port/events.log}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1545) Topology Debug Event Log in Wrong Location
[
https://issues.apache.org/jira/browse/STORM-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15156537#comment-15156537
]
ASF GitHub Bot commented on STORM-1545:
---
Github user abhishekagarwal87 commented on the pull request:
https://github.com/apache/storm/pull/1123#issuecomment-187033903
Thanks @mproch Looks good to me (NB) for merge with master.
> Topology Debug Event Log in Wrong Location
> --
>
> Key: STORM-1545
> URL: https://issues.apache.org/jira/browse/STORM-1545
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core, storm-ui
>Affects Versions: 1.0.0
>Reporter: P. Taylor Goetz
> Labels: newbie
>
> Currently the {{events.log}} file is not created where the log viewer expects
> it so be, so the "events" link under debug in Storm UI returns a 404.
> The file should be in:
> {{$storm.log.dir/workers-artifacts/$topology_id/$port/events.log}}
> but is instead in:
> {{$storm.local.dir/workers-artifacts/$topology_id/$port/events.log}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: closes STORM-1545 and STORM-1552
Github user abhishekagarwal87 commented on the pull request: https://github.com/apache/storm/pull/1123#issuecomment-187033903 Thanks @mproch Looks good to me (NB) for merge with master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1566) Worker exits with error o.a.s.d.worker [ERROR] Error on initialization of server mk-worker java.lang.ClassCastException: java.lang.String cannot be cast to java.io.File
[ https://issues.apache.org/jira/browse/STORM-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15156541#comment-15156541 ] ASF GitHub Bot commented on STORM-1566: --- GitHub user satishd opened a pull request: https://github.com/apache/storm/pull/1133 STORM-1566 Passing File instance instead of path String You can merge this pull request into a Git repository by running: $ git pull https://github.com/satishd/storm STORM-1566 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1133.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1133 commit ef6dfb9943163d80ff1355e6e1cfcac977d2b8cd Author: Satish Duggana Date: 2016-02-22T06:32:26Z STORM-1566 Passing File instance of path String > Worker exits with error o.a.s.d.worker [ERROR] Error on initialization of > server mk-worker java.lang.ClassCastException: java.lang.String cannot be > cast to java.io.File > > > Key: STORM-1566 > URL: https://issues.apache.org/jira/browse/STORM-1566 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 2.0.0 >Reporter: Satish Duggana >Assignee: Satish Duggana > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1566 Passing File instance instead of pa...
GitHub user satishd opened a pull request: https://github.com/apache/storm/pull/1133 STORM-1566 Passing File instance instead of path String You can merge this pull request into a Git repository by running: $ git pull https://github.com/satishd/storm STORM-1566 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1133.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1133 commit ef6dfb9943163d80ff1355e6e1cfcac977d2b8cd Author: Satish Duggana Date: 2016-02-22T06:32:26Z STORM-1566 Passing File instance of path String --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: storm-kafka : kafka-partition can not find lea...
Github user abhishekagarwal87 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1132#discussion_r53590435
--- Diff:
external/storm-kafka/src/jvm/org/apache/storm/kafka/DynamicBrokersReader.java
---
@@ -85,6 +85,10 @@ public DynamicBrokersReader(Map conf, String zkStr,
String zkPath, String topic)
String brokerInfoPath = brokerPath();
for (int partition = 0; partition < numPartitionsForTopic;
partition++) {
int leader = getLeaderFor(topic,partition);
+ if(leader == -1){
--- End diff --
indentation is off here. Please add space around curly brace.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: storm-kafka : kafka-partition can not find lea...
Github user abhishekagarwal87 commented on the pull request: https://github.com/apache/storm/pull/1132#issuecomment-187040728 I am thinking of the side effects that this change can lead to - 1. Number of tasks could be more than number of partitions - Acceptable and will be corrected in next refresh 2. GlobalPartitionMapping may be different for multiple tasks - Depending on when the spout task reads the kafka cluster zookeeper, it is possible that task A reads 15 partitions and task B reads 16 partitions for a topic with total 16 partitions. This can be problematic as it will lead to two tasks assigning same partitions to themselves. It breaks the contract that same partition won't be read by two different tasks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-1566 Passing File instance instead of pa...
Github user abhishekagarwal87 commented on the pull request: https://github.com/apache/storm/pull/1133#issuecomment-187043365 +1 NB --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1566) Worker exits with error o.a.s.d.worker [ERROR] Error on initialization of server mk-worker java.lang.ClassCastException: java.lang.String cannot be cast to java.io.File
[ https://issues.apache.org/jira/browse/STORM-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15156570#comment-15156570 ] ASF GitHub Bot commented on STORM-1566: --- Github user abhishekagarwal87 commented on the pull request: https://github.com/apache/storm/pull/1133#issuecomment-187043365 +1 NB > Worker exits with error o.a.s.d.worker [ERROR] Error on initialization of > server mk-worker java.lang.ClassCastException: java.lang.String cannot be > cast to java.io.File > > > Key: STORM-1566 > URL: https://issues.apache.org/jira/browse/STORM-1566 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 2.0.0 >Reporter: Satish Duggana >Assignee: Satish Duggana > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-917 : Ability to emit from the spout to ...
Github user anishek commented on the pull request: https://github.com/apache/storm/pull/601#issuecomment-187051503 @lujinhong - i have not worked with trident and hence don't know where to make the change. @abhishekagarwal87 -- this was a quick fix i did for a POC under 30 mins, It might not be the right way to go about this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-917 : Ability to emit from the spout to ...
Github user anishek commented on a diff in the pull request: https://github.com/apache/storm/pull/601#discussion_r53592727 --- Diff: external/storm-kafka/src/jvm/storm/kafka/SpoutConfig.java --- @@ -17,6 +17,8 @@ */ package storm.kafka; +import backtype.storm.utils.Utils; --- End diff -- Upmerge ? i thought one of you would pull in the request and merge it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-917) Ability to emit messages from a kafka spout to a specific stream
[ https://issues.apache.org/jira/browse/STORM-917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15156580#comment-15156580 ] ASF GitHub Bot commented on STORM-917: -- Github user anishek commented on a diff in the pull request: https://github.com/apache/storm/pull/601#discussion_r53592727 --- Diff: external/storm-kafka/src/jvm/storm/kafka/SpoutConfig.java --- @@ -17,6 +17,8 @@ */ package storm.kafka; +import backtype.storm.utils.Utils; --- End diff -- Upmerge ? i thought one of you would pull in the request and merge it. > Ability to emit messages from a kafka spout to a specific stream > - > > Key: STORM-917 > URL: https://issues.apache.org/jira/browse/STORM-917 > Project: Apache Storm > Issue Type: Improvement > Components: storm-kafka >Reporter: anishek > Fix For: 1.0.0 > > > Today even if we have multiple spouts in storm they will all emit on the same > "default" stream. Having an ability to configure the stream that a kafka > spout emits to would be great. By default should emit to the "default" stream -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-917) Ability to emit messages from a kafka spout to a specific stream
[ https://issues.apache.org/jira/browse/STORM-917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15156581#comment-15156581 ] ASF GitHub Bot commented on STORM-917: -- Github user anishek commented on the pull request: https://github.com/apache/storm/pull/601#issuecomment-187051503 @lujinhong - i have not worked with trident and hence don't know where to make the change. @abhishekagarwal87 -- this was a quick fix i did for a POC under 30 mins, It might not be the right way to go about this. > Ability to emit messages from a kafka spout to a specific stream > - > > Key: STORM-917 > URL: https://issues.apache.org/jira/browse/STORM-917 > Project: Apache Storm > Issue Type: Improvement > Components: storm-kafka >Reporter: anishek > Fix For: 1.0.0 > > > Today even if we have multiple spouts in storm they will all emit on the same > "default" stream. Having an ability to configure the stream that a kafka > spout emits to would be great. By default should emit to the "default" stream -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-917 : Ability to emit from the spout to ...
Github user abhishekagarwal87 commented on a diff in the pull request: https://github.com/apache/storm/pull/601#discussion_r53593326 --- Diff: external/storm-kafka/src/jvm/storm/kafka/SpoutConfig.java --- @@ -17,6 +17,8 @@ */ package storm.kafka; +import backtype.storm.utils.Utils; --- End diff -- One of the committers will do that. But for that to happen, the PR should be ready for merge. Currently it has conflicts with master branch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-917) Ability to emit messages from a kafka spout to a specific stream
[ https://issues.apache.org/jira/browse/STORM-917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15156586#comment-15156586 ] ASF GitHub Bot commented on STORM-917: -- Github user abhishekagarwal87 commented on a diff in the pull request: https://github.com/apache/storm/pull/601#discussion_r53593326 --- Diff: external/storm-kafka/src/jvm/storm/kafka/SpoutConfig.java --- @@ -17,6 +17,8 @@ */ package storm.kafka; +import backtype.storm.utils.Utils; --- End diff -- One of the committers will do that. But for that to happen, the PR should be ready for merge. Currently it has conflicts with master branch. > Ability to emit messages from a kafka spout to a specific stream > - > > Key: STORM-917 > URL: https://issues.apache.org/jira/browse/STORM-917 > Project: Apache Storm > Issue Type: Improvement > Components: storm-kafka >Reporter: anishek > Fix For: 1.0.0 > > > Today even if we have multiple spouts in storm they will all emit on the same > "default" stream. Having an ability to configure the stream that a kafka > spout emits to would be great. By default should emit to the "default" stream -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (STORM-1244) port backtype.storm.command.upload-credentials to java
[ https://issues.apache.org/jira/browse/STORM-1244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Agarwal reassigned STORM-1244: --- Assignee: Abhishek Agarwal > port backtype.storm.command.upload-credentials to java > -- > > Key: STORM-1244 > URL: https://issues.apache.org/jira/browse/STORM-1244 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: Abhishek Agarwal > Labels: java-migration, jstorm-merger > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (STORM-1290) port backtype.storm.local-state-test to java
[ https://issues.apache.org/jira/browse/STORM-1290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Agarwal reassigned STORM-1290: --- Assignee: Abhishek Agarwal > port backtype.storm.local-state-test to java > - > > Key: STORM-1290 > URL: https://issues.apache.org/jira/browse/STORM-1290 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: Abhishek Agarwal > Labels: java-migration, jstorm-merger > > test for LocalState class -- This message was sent by Atlassian JIRA (v6.3.4#6332)
