date:20160313

[jira] [Commented] (STORM-822) As a storm developer I’d like to use the new kafka consumer API (0.8.3) to reduce dependencies and use long term supported kafka apis

2016-03-13 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/STORM-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15192754#comment-15192754
 ] 

ASF GitHub Bot commented on STORM-822:
--

Github user abhishekagarwal87 commented on a diff in the pull request:

https://github.com/apache/storm/pull/1131#discussion_r55957034
  
--- Diff: 
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutStreams.java
 ---
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ *   or more contributor license agreements.  See the NOTICE file
+ *   distributed with this work for additional information
+ *   regarding copyright ownership.  The ASF licenses this file
+ *   to you under the Apache License, Version 2.0 (the
+ *   "License"); you may not use this file except in compliance
+ *   with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *   Unless required by applicable law or agreed to in writing, software
+ *   distributed under the License is distributed on an "AS IS" BASIS,
+ *   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *   See the License for the specific language governing permissions and
+ *   limitations under the License.
+ */
+
+package org.apache.storm.kafka.spout;
+
+import org.apache.storm.spout.SpoutOutputCollector;
+import org.apache.storm.topology.OutputFieldsDeclarer;
+import org.apache.storm.tuple.Fields;
+import org.apache.storm.utils.Utils;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Represents the output streams associated with each topic, and provides 
a public API to
+ * declare output streams and emmit tuples, on the appropriate stream, for 
all the topics specified.
+ */
+public class KafkaSpoutStreams implements Serializable {
+private final Map topicToStream;
+
+private KafkaSpoutStreams(Builder builder) {
+this.topicToStream = builder.topicToStream;
+}
+
+/**
+ * @param topic the topic for which to get output fields
+ * @return the output fields declared
+ */
+public Fields getOutputFields(String topic) {
+if (topicToStream.containsKey(topic)) {
+return topicToStream.get(topic).getOutputFields();
+}
+throw new IllegalStateException(this.getClass().getName() + " not 
configured for topic: " + topic);
+}
+
+/**
+ * @param topic the topic to for which to get the stream id
+ * @return the id of the stream to where the tuples are emitted
+ */
+public String getStreamId(String topic) {
+if (topicToStream.containsKey(topic)) {
+return topicToStream.get(topic).getStreamId();
+}
+throw new IllegalStateException(this.getClass().getName() + " not 
configured for topic: " + topic);
+}
+
+/**
+ * @return list of topics subscribed and emitting tuples to a stream 
as configured by {@link KafkaSpoutStream}
+ */
+public List getTopics() {
+return new ArrayList<>(topicToStream.keySet());
+}
+
+void declareOutputFields(OutputFieldsDeclarer declarer) {
+for (KafkaSpoutStream stream : topicToStream.values()) {
+declarer.declareStream(stream.getStreamId(), 
stream.getOutputFields());
+}
+}
+
+void emit(SpoutOutputCollector collector, MessageId messageId) {
+collector.emit(getStreamId(messageId.topic()), 
messageId.getTuple(), messageId);
--- End diff --

older consumer didn't call the fetch again till the _waitingToEmit queue 
became empty. And records were only stored in _waitingToEmit. Here however, 
they are also stored as a part of message id. The message id in turn is stored 
in two data structures - 
1. pending map of spout - It has an upper ceiling of 
topology.max.spout.pending * sizeof(message id) bytes
2. acked map - This stores the message ids that have been acked. It is 
cleared on a time basis and doesn't have any upper ceiling on number of entries 
per TopicPartition.
@hmcl @harshach Please let me know if above two are valid inferences. 
Nonetheless, I am +1 on resolving this discussion later without blocking the 
PR. 


> As a storm developer I’d like to use the new kafka consumer API (0.8.3) to 
> reduce dependencies and use long term supported kafka apis 
> --
>
> Key: STORM-8

[GitHub] storm pull request: STORM-822: Kafka Spout New Consumer API

2016-03-13 Thread abhishekagarwal87

Github user abhishekagarwal87 commented on a diff in the pull request:

https://github.com/apache/storm/pull/1131#discussion_r55957034
  
--- Diff: 
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutStreams.java
 ---
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ *   or more contributor license agreements.  See the NOTICE file
+ *   distributed with this work for additional information
+ *   regarding copyright ownership.  The ASF licenses this file
+ *   to you under the Apache License, Version 2.0 (the
+ *   "License"); you may not use this file except in compliance
+ *   with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *   Unless required by applicable law or agreed to in writing, software
+ *   distributed under the License is distributed on an "AS IS" BASIS,
+ *   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *   See the License for the specific language governing permissions and
+ *   limitations under the License.
+ */
+
+package org.apache.storm.kafka.spout;
+
+import org.apache.storm.spout.SpoutOutputCollector;
+import org.apache.storm.topology.OutputFieldsDeclarer;
+import org.apache.storm.tuple.Fields;
+import org.apache.storm.utils.Utils;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Represents the output streams associated with each topic, and provides 
a public API to
+ * declare output streams and emmit tuples, on the appropriate stream, for 
all the topics specified.
+ */
+public class KafkaSpoutStreams implements Serializable {
+private final Map topicToStream;
+
+private KafkaSpoutStreams(Builder builder) {
+this.topicToStream = builder.topicToStream;
+}
+
+/**
+ * @param topic the topic for which to get output fields
+ * @return the output fields declared
+ */
+public Fields getOutputFields(String topic) {
+if (topicToStream.containsKey(topic)) {
+return topicToStream.get(topic).getOutputFields();
+}
+throw new IllegalStateException(this.getClass().getName() + " not 
configured for topic: " + topic);
+}
+
+/**
+ * @param topic the topic to for which to get the stream id
+ * @return the id of the stream to where the tuples are emitted
+ */
+public String getStreamId(String topic) {
+if (topicToStream.containsKey(topic)) {
+return topicToStream.get(topic).getStreamId();
+}
+throw new IllegalStateException(this.getClass().getName() + " not 
configured for topic: " + topic);
+}
+
+/**
+ * @return list of topics subscribed and emitting tuples to a stream 
as configured by {@link KafkaSpoutStream}
+ */
+public List getTopics() {
+return new ArrayList<>(topicToStream.keySet());
+}
+
+void declareOutputFields(OutputFieldsDeclarer declarer) {
+for (KafkaSpoutStream stream : topicToStream.values()) {
+declarer.declareStream(stream.getStreamId(), 
stream.getOutputFields());
+}
+}
+
+void emit(SpoutOutputCollector collector, MessageId messageId) {
+collector.emit(getStreamId(messageId.topic()), 
messageId.getTuple(), messageId);
--- End diff --

older consumer didn't call the fetch again till the _waitingToEmit queue 
became empty. And records were only stored in _waitingToEmit. Here however, 
they are also stored as a part of message id. The message id in turn is stored 
in two data structures - 
1. pending map of spout - It has an upper ceiling of 
topology.max.spout.pending * sizeof(message id) bytes
2. acked map - This stores the message ids that have been acked. It is 
cleared on a time basis and doesn't have any upper ceiling on number of entries 
per TopicPartition.
@hmcl @harshach Please let me know if above two are valid inferences. 
Nonetheless, I am +1 on resolving this discussion later without blocking the 
PR. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Different topologies need different schedulers

2016-03-13 Thread devopts

Hi all,
I think it is a issue that all topologies only is scheduled by the same 
scheduler.
And it doesn't work when i set the "storm.scheduler" value in my topology,such 
as 
config.put(Config.STORM_SCHEDULE,xxx.xxx).
Now ,different topologies needs to be scheduled by different schedulers.
And how to solve the problem?

Re: Issues in Storm deployment

2016-03-13 Thread vibha goyal

It worked!!

Can you please help me about how to change the storm.yaml file ?


Thanks already!!


On Sun, Mar 13, 2016 at 5:55 PM, vibha goyal  wrote:

> Hi ,
>
> Thanks for prompt reply.
>
> But, I have to use on multi-node cluster, and changes in storm.yaml will
> be required.
>
> Will the below settings work?
>
>
>
> On Sun, Mar 13, 2016 at 4:15 PM, Harsha  wrote:
>
>>  Vibhay,
>>  You are using master branch which is under development to
>>  move to Java. This might be very unstable. I suggest you to
>>  use 1.x-branch in general as its getting ready for storm
>>  1.0 release. You don't need to add anything to storm.yaml ,
>>  defaults should be fine for a single node cluster.
>>
>> 1. git checkout 1.x-branch
>> 2. mvn -DskipTests clean install
>> 3. cd storm-dist/binary
>> 4. mvn clean package
>> 5. cd target
>> 6. find and unzip apache-storm-1.0.0-SNAPSHOT
>> 7. run required services
>>   bin/storm dev-zookeeper, bin/storm nimbus, bin/storm supervisor,
>>   bin/storm ui
>> 8. ./bin/storm jar
>> examples/storm-starter/storm-starter-topologies-1.0.0-SNAPSHOT.jar
>> org.apache.storm.starter.WordCountTopology wordcount
>>
>> For bit more details you can refer here
>> http://blog.harsha.io/setting-up-a-single-node-apache-storm-cluster/
>> its on a older version of storm but relevant for the latest as well.
>>
>> -Harsha
>>
>>
>>
>>
>> On Sun, Mar 13, 2016, at 01:43 PM, vibha goyal wrote:
>> > Hi,
>> >
>> > I am grad student, and I am working on Storm as a part of my course
>> > project.
>> >
>> > I have cloned the source code from git, and followed the instructions.
>> >
>> > I am using branch : 0.10.0
>> >
>> > After "mvn package", when I copy the apache-storm-.tar.gz in my
>> > home,
>> > untar it and try to run nimbus.
>> >
>> > I get error:
>> >
>> > Exception in thread "main" java.lang.ExceptionInInitializerError
>> > at
>> >
>> org.apache.storm.utils.ConfigUtils.readStormConfigImpl(ConfigUtils.java:138)
>> > at
>> > org.apache.storm.utils.ConfigUtils.readStormConfig(ConfigUtils.java:134)
>> > at org.apache.storm.command.ConfigValue.main(ConfigValue.java:27)
>> > Caused by: java.lang.RuntimeException: java.io.IOException: Found
>> > multiple
>> > storm.yaml resources. You're probably bundling the Storm jars with your
>> > topology jar.
>> >
>> [jar:file:/home/vgoyal5/apache-storm-2.0.0-SNAPSHOT/lib/storm-core-2.0.0-SNAPSHOT.jar!/storm.yaml,
>> > file:/home/vgoyal5/apache-storm-2.0.0-SNAPSHOT/conf/storm.yaml]
>> > at org.apache.storm.utils.Utils.findAndReadConfigFile(Utils.java:372)
>> > at org.apache.storm.utils.Utils.readStormConfig(Utils.java:456)
>> > at org.apache.storm.utils.Utils.(Utils.java:173)
>> > ... 3 more
>> >
>> >
>> > I deleted the
>> "/home/vgoyal5/apache-storm-2.0.0-SNAPSHOT/conf/storm.yaml"
>> > file and again tried.
>> >
>> > This time I was able to connect the nimbus and supervisors and could see
>> > the connections in UI.
>> >
>> > As soon as I tried to submit the topology with the command:
>> >
>> > storm jar examples/storm-starter/target/storm-starter-2.0.0-SNAPSHOT.jar
>> > org.apache.storm.starter.ExclamationTopology exclamation-topology
>> >
>> > I got error:
>> >
>> > org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find
>> > leader
>> > nimbus from seed hosts [sp16-cs525-g05-01.cs.illinois.edu]. Did you
>> > specify
>> > a valid list of nimbus hosts for config nimbus.seeds?
>> >
>> > First ,I am not able to resolve this error,
>> >
>> > and second, I cannot make changes in storm.yaml as I deleted the
>> > conf/storm.yaml
>> >
>> > I am a beginner in Storm. I have been trying this for a day now. Any
>> help
>> > would be appreciated!!
>> >
>> >
>> >
>> > Thanks!
>> >
>> > Vibha Goyal
>>
>
>

Re: Issues in Storm deployment

2016-03-13 Thread vibha goyal

Hi ,

Thanks for prompt reply.

But, I have to use on multi-node cluster, and changes in storm.yaml will be
required.

Will the below settings work?



On Sun, Mar 13, 2016 at 4:15 PM, Harsha  wrote:

>  Vibhay,
>  You are using master branch which is under development to
>  move to Java. This might be very unstable. I suggest you to
>  use 1.x-branch in general as its getting ready for storm
>  1.0 release. You don't need to add anything to storm.yaml ,
>  defaults should be fine for a single node cluster.
>
> 1. git checkout 1.x-branch
> 2. mvn -DskipTests clean install
> 3. cd storm-dist/binary
> 4. mvn clean package
> 5. cd target
> 6. find and unzip apache-storm-1.0.0-SNAPSHOT
> 7. run required services
>   bin/storm dev-zookeeper, bin/storm nimbus, bin/storm supervisor,
>   bin/storm ui
> 8. ./bin/storm jar
> examples/storm-starter/storm-starter-topologies-1.0.0-SNAPSHOT.jar
> org.apache.storm.starter.WordCountTopology wordcount
>
> For bit more details you can refer here
> http://blog.harsha.io/setting-up-a-single-node-apache-storm-cluster/
> its on a older version of storm but relevant for the latest as well.
>
> -Harsha
>
>
>
>
> On Sun, Mar 13, 2016, at 01:43 PM, vibha goyal wrote:
> > Hi,
> >
> > I am grad student, and I am working on Storm as a part of my course
> > project.
> >
> > I have cloned the source code from git, and followed the instructions.
> >
> > I am using branch : 0.10.0
> >
> > After "mvn package", when I copy the apache-storm-.tar.gz in my
> > home,
> > untar it and try to run nimbus.
> >
> > I get error:
> >
> > Exception in thread "main" java.lang.ExceptionInInitializerError
> > at
> >
> org.apache.storm.utils.ConfigUtils.readStormConfigImpl(ConfigUtils.java:138)
> > at
> > org.apache.storm.utils.ConfigUtils.readStormConfig(ConfigUtils.java:134)
> > at org.apache.storm.command.ConfigValue.main(ConfigValue.java:27)
> > Caused by: java.lang.RuntimeException: java.io.IOException: Found
> > multiple
> > storm.yaml resources. You're probably bundling the Storm jars with your
> > topology jar.
> >
> [jar:file:/home/vgoyal5/apache-storm-2.0.0-SNAPSHOT/lib/storm-core-2.0.0-SNAPSHOT.jar!/storm.yaml,
> > file:/home/vgoyal5/apache-storm-2.0.0-SNAPSHOT/conf/storm.yaml]
> > at org.apache.storm.utils.Utils.findAndReadConfigFile(Utils.java:372)
> > at org.apache.storm.utils.Utils.readStormConfig(Utils.java:456)
> > at org.apache.storm.utils.Utils.(Utils.java:173)
> > ... 3 more
> >
> >
> > I deleted the "/home/vgoyal5/apache-storm-2.0.0-SNAPSHOT/conf/storm.yaml"
> > file and again tried.
> >
> > This time I was able to connect the nimbus and supervisors and could see
> > the connections in UI.
> >
> > As soon as I tried to submit the topology with the command:
> >
> > storm jar examples/storm-starter/target/storm-starter-2.0.0-SNAPSHOT.jar
> > org.apache.storm.starter.ExclamationTopology exclamation-topology
> >
> > I got error:
> >
> > org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find
> > leader
> > nimbus from seed hosts [sp16-cs525-g05-01.cs.illinois.edu]. Did you
> > specify
> > a valid list of nimbus hosts for config nimbus.seeds?
> >
> > First ,I am not able to resolve this error,
> >
> > and second, I cannot make changes in storm.yaml as I deleted the
> > conf/storm.yaml
> >
> > I am a beginner in Storm. I have been trying this for a day now. Any help
> > would be appreciated!!
> >
> >
> >
> > Thanks!
> >
> > Vibha Goyal
>

[jira] [Commented] (STORM-1235) port backtype.storm.security.auth.ReqContext-test to java

2016-03-13 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/STORM-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15192534#comment-15192534
 ] 

ASF GitHub Bot commented on STORM-1235:
---

Github user harshach commented on the pull request:

https://github.com/apache/storm/pull/1200#issuecomment-196055599
  
@abhishekagarwal87 ok makes sense confused by the PR title.


> port  backtype.storm.security.auth.ReqContext-test to java
> --
>
> Key: STORM-1235
> URL: https://issues.apache.org/jira/browse/STORM-1235
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Abhishek Agarwal
>  Labels: java-migration, jstorm-merger
>
> junit test migration



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: Tests - STORM-1235, STORM-1236, STORM-1237, ST...

2016-03-13 Thread harshach

Github user harshach commented on the pull request:

https://github.com/apache/storm/pull/1200#issuecomment-196055599
  
@abhishekagarwal87 ok makes sense confused by the PR title.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Issues in Storm deployment

2016-03-13 Thread Harsha

 Vibhay,
 You are using master branch which is under development to
 move to Java. This might be very unstable. I suggest you to
 use 1.x-branch in general as its getting ready for storm
 1.0 release. You don't need to add anything to storm.yaml ,
 defaults should be fine for a single node cluster.

1. git checkout 1.x-branch
2. mvn -DskipTests clean install
3. cd storm-dist/binary
4. mvn clean package
5. cd target
6. find and unzip apache-storm-1.0.0-SNAPSHOT
7. run required services
  bin/storm dev-zookeeper, bin/storm nimbus, bin/storm supervisor,
  bin/storm ui
8. ./bin/storm jar
examples/storm-starter/storm-starter-topologies-1.0.0-SNAPSHOT.jar
org.apache.storm.starter.WordCountTopology wordcount

For bit more details you can refer here
http://blog.harsha.io/setting-up-a-single-node-apache-storm-cluster/
its on a older version of storm but relevant for the latest as well.

-Harsha




On Sun, Mar 13, 2016, at 01:43 PM, vibha goyal wrote:
> Hi,
> 
> I am grad student, and I am working on Storm as a part of my course
> project.
> 
> I have cloned the source code from git, and followed the instructions.
> 
> I am using branch : 0.10.0
> 
> After "mvn package", when I copy the apache-storm-.tar.gz in my
> home,
> untar it and try to run nimbus.
> 
> I get error:
> 
> Exception in thread "main" java.lang.ExceptionInInitializerError
> at
> org.apache.storm.utils.ConfigUtils.readStormConfigImpl(ConfigUtils.java:138)
> at
> org.apache.storm.utils.ConfigUtils.readStormConfig(ConfigUtils.java:134)
> at org.apache.storm.command.ConfigValue.main(ConfigValue.java:27)
> Caused by: java.lang.RuntimeException: java.io.IOException: Found
> multiple
> storm.yaml resources. You're probably bundling the Storm jars with your
> topology jar.
> [jar:file:/home/vgoyal5/apache-storm-2.0.0-SNAPSHOT/lib/storm-core-2.0.0-SNAPSHOT.jar!/storm.yaml,
> file:/home/vgoyal5/apache-storm-2.0.0-SNAPSHOT/conf/storm.yaml]
> at org.apache.storm.utils.Utils.findAndReadConfigFile(Utils.java:372)
> at org.apache.storm.utils.Utils.readStormConfig(Utils.java:456)
> at org.apache.storm.utils.Utils.(Utils.java:173)
> ... 3 more
> 
> 
> I deleted the "/home/vgoyal5/apache-storm-2.0.0-SNAPSHOT/conf/storm.yaml"
> file and again tried.
> 
> This time I was able to connect the nimbus and supervisors and could see
> the connections in UI.
> 
> As soon as I tried to submit the topology with the command:
> 
> storm jar examples/storm-starter/target/storm-starter-2.0.0-SNAPSHOT.jar
> org.apache.storm.starter.ExclamationTopology exclamation-topology
> 
> I got error:
> 
> org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find
> leader
> nimbus from seed hosts [sp16-cs525-g05-01.cs.illinois.edu]. Did you
> specify
> a valid list of nimbus hosts for config nimbus.seeds?
> 
> First ,I am not able to resolve this error,
> 
> and second, I cannot make changes in storm.yaml as I deleted the
> conf/storm.yaml
> 
> I am a beginner in Storm. I have been trying this for a day now. Any help
> would be appreciated!!
> 
> 
> 
> Thanks!
> 
> Vibha Goyal

Issues in Storm deployment

2016-03-13 Thread vibha goyal

Hi,

I am grad student, and I am working on Storm as a part of my course project.

I have cloned the source code from git, and followed the instructions.

I am using branch : 0.10.0

After "mvn package", when I copy the apache-storm-.tar.gz in my
home,
untar it and try to run nimbus.

I get error:

Exception in thread "main" java.lang.ExceptionInInitializerError
at
org.apache.storm.utils.ConfigUtils.readStormConfigImpl(ConfigUtils.java:138)
at org.apache.storm.utils.ConfigUtils.readStormConfig(ConfigUtils.java:134)
at org.apache.storm.command.ConfigValue.main(ConfigValue.java:27)
Caused by: java.lang.RuntimeException: java.io.IOException: Found multiple
storm.yaml resources. You're probably bundling the Storm jars with your
topology jar.
[jar:file:/home/vgoyal5/apache-storm-2.0.0-SNAPSHOT/lib/storm-core-2.0.0-SNAPSHOT.jar!/storm.yaml,
file:/home/vgoyal5/apache-storm-2.0.0-SNAPSHOT/conf/storm.yaml]
at org.apache.storm.utils.Utils.findAndReadConfigFile(Utils.java:372)
at org.apache.storm.utils.Utils.readStormConfig(Utils.java:456)
at org.apache.storm.utils.Utils.(Utils.java:173)
... 3 more

I deleted the "/home/vgoyal5/apache-storm-2.0.0-SNAPSHOT/conf/storm.yaml"
file and again tried.

This time I was able to connect the nimbus and supervisors and could see
the connections in UI.

As soon as I tried to submit the topology with the command:

storm jar examples/storm-starter/target/storm-starter-2.0.0-SNAPSHOT.jar
org.apache.storm.starter.ExclamationTopology exclamation-topology

I got error:

org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find leader
nimbus from seed hosts [sp16-cs525-g05-01.cs.illinois.edu]. Did you specify
a valid list of nimbus hosts for config nimbus.seeds?

86 matches

Mail list logo