[jira] [Updated] (STORM-1404) Fix Mockito test failures in storm-kafka
[
https://issues.apache.org/jira/browse/STORM-1404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated STORM-1404:
--
Summary: Fix Mockito test failures in storm-kafka (was: Mockito test
failures in storm-kafka)
> Fix Mockito test failures in storm-kafka
>
>
> Key: STORM-1404
> URL: https://issues.apache.org/jira/browse/STORM-1404
> Project: Apache Storm
> Issue Type: Sub-task
> Components: storm-core
>Reporter: Derek Dagit
>Assignee: Haohui Mai
>Priority: Minor
>
> {noformat}
> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 17.194 sec
> <<< FAILURE! - in storm.kafka.bolt.KafkaBoltTest
> executeWithByteArrayKeyAndMessageAsync(storm.kafka.bolt.KafkaBoltTest) Time
> elapsed: 3.526 sec <<< FAILURE!
> org.mockito.exceptions.verification.WantedButNotInvoked:
> Wanted but not invoked:
> collector.ack(
> source: null:1, stream: , id: {}, [[B@29e2aabd, [B@603773c6]
> );
> -> at
> storm.kafka.bolt.KafkaBoltTest.executeWithByteArrayKeyAndMessageAsync(KafkaBoltTest.java:144)
> However, there were other interactions with this mock:
> -> at
> backtype.storm.task.OutputCollector.reportError(OutputCollector.java:223)
> -> at backtype.storm.task.OutputCollector.fail(OutputCollector.java:218)
> at
> storm.kafka.bolt.KafkaBoltTest.executeWithByteArrayKeyAndMessageAsync(KafkaBoltTest.java:144)
> {noformat}
> Seen [here|https://travis-ci.org/apache/storm/jobs/97734764#L352]
> Not sure about how often this happens, but it is definitely not 100% of the
> time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-822) As a storm developer I’d like to use the new kafka consumer API (0.8.3) to reduce dependencies and use long term supported kafka apis
[ https://issues.apache.org/jira/browse/STORM-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088785#comment-15088785 ] ASF GitHub Bot commented on STORM-822: -- Github user hmcl commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-169910619 @Deepnekroz I am working on this. We wil make sure that all works well with what you have. > As a storm developer I’d like to use the new kafka consumer API (0.8.3) to > reduce dependencies and use long term supported kafka apis > -- > > Key: STORM-822 > URL: https://issues.apache.org/jira/browse/STORM-822 > Project: Apache Storm > Issue Type: Story > Components: storm-kafka >Reporter: Thomas Becker >Assignee: Hugo Louro > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user hmcl commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-169910619 @Deepnekroz I am working on this. We wil make sure that all works well with what you have. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-1406: Add MQTT Support
Github user arunmahadevan commented on a diff in the pull request:
https://github.com/apache/storm/pull/991#discussion_r49161519
--- Diff:
external/storm-mqtt/core/src/main/java/org/apache/storm/mqtt/trident/MqttPublishFunction.java
---
@@ -0,0 +1,84 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.mqtt.trident;
+
+import backtype.storm.Config;
+import backtype.storm.task.OutputCollector;
+import backtype.storm.topology.FailedException;
+import org.apache.storm.mqtt.MqttMessage;
+import org.apache.storm.mqtt.common.MqttOptions;
+import org.apache.storm.mqtt.MqttTupleMapper;
+import org.apache.storm.mqtt.common.MqttPublisher;
+import org.apache.storm.mqtt.common.SslUtils;
+import org.apache.storm.mqtt.ssl.KeyStoreLoader;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import storm.trident.operation.BaseFunction;
+import storm.trident.operation.TridentCollector;
+import storm.trident.operation.TridentOperationContext;
+import storm.trident.tuple.TridentTuple;
+
+import java.util.Map;
+
+public class MqttPublishFunction extends BaseFunction {
--- End diff --
Since the function does not emit anything, it cannot be chained with
further operations like normal functions and it appeared more like a sink.
Hence thought it might be more apt for it to be implemented as a state instead
of a function.
Agree that its difficult to do exactly once, but many other state
implementations also does not guarantee exactly once. Leave it to you to decide
the right approach.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-1406) MQTT Support
[
https://issues.apache.org/jira/browse/STORM-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088777#comment-15088777
]
ASF GitHub Bot commented on STORM-1406:
---
Github user arunmahadevan commented on a diff in the pull request:
https://github.com/apache/storm/pull/991#discussion_r49161519
--- Diff:
external/storm-mqtt/core/src/main/java/org/apache/storm/mqtt/trident/MqttPublishFunction.java
---
@@ -0,0 +1,84 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.mqtt.trident;
+
+import backtype.storm.Config;
+import backtype.storm.task.OutputCollector;
+import backtype.storm.topology.FailedException;
+import org.apache.storm.mqtt.MqttMessage;
+import org.apache.storm.mqtt.common.MqttOptions;
+import org.apache.storm.mqtt.MqttTupleMapper;
+import org.apache.storm.mqtt.common.MqttPublisher;
+import org.apache.storm.mqtt.common.SslUtils;
+import org.apache.storm.mqtt.ssl.KeyStoreLoader;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import storm.trident.operation.BaseFunction;
+import storm.trident.operation.TridentCollector;
+import storm.trident.operation.TridentOperationContext;
+import storm.trident.tuple.TridentTuple;
+
+import java.util.Map;
+
+public class MqttPublishFunction extends BaseFunction {
--- End diff --
Since the function does not emit anything, it cannot be chained with
further operations like normal functions and it appeared more like a sink.
Hence thought it might be more apt for it to be implemented as a state instead
of a function.
Agree that its difficult to do exactly once, but many other state
implementations also does not guarantee exactly once. Leave it to you to decide
the right approach.
> MQTT Support
>
>
> Key: STORM-1406
> URL: https://issues.apache.org/jira/browse/STORM-1406
> Project: Apache Storm
> Issue Type: Improvement
>Reporter: P. Taylor Goetz
>Assignee: P. Taylor Goetz
>
> MQTT is a lightweight publish/subscribe protocol frequently used in IoT
> applications.
> Further information can be found at http://mqtt.org
> Initial features include:
> * Full MQTT support (e.g. last will, QoS 0-2, retain, etc.)
> * Spout implementation(s) for subscribing to MQTT topics
> * A bolt implementation for publishing MQTT messages
> * A trident function implementation for publishing MQTT messages
> * Authentication and TLS/SSL support
> * User-defined "mappers" for converting MQTT messages to tuples (subscribers)
> * User-defined "mappers" for converting tuples to MQTT messages (publishers)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-822) As a storm developer I’d like to use the new kafka consumer API (0.8.3) to reduce dependencies and use long term supported kafka apis
[ https://issues.apache.org/jira/browse/STORM-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088768#comment-15088768 ] ASF GitHub Bot commented on STORM-822: -- Github user Deepnekroz commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-169908193 Thanks for comments! @harshach could you please give me ETA of @hmcl PR with this feature? I want to be sure, that new storm-kafka will work nicely with my existing code base. Also, I can write non-autocommit approach by next two days, if its needed. Have a nice day! > As a storm developer I’d like to use the new kafka consumer API (0.8.3) to > reduce dependencies and use long term supported kafka apis > -- > > Key: STORM-822 > URL: https://issues.apache.org/jira/browse/STORM-822 > Project: Apache Storm > Issue Type: Story > Components: storm-kafka >Reporter: Thomas Becker >Assignee: Hugo Louro > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user Deepnekroz commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-169908193 Thanks for comments! @harshach could you please give me ETA of @hmcl PR with this feature? I want to be sure, that new storm-kafka will work nicely with my existing code base. Also, I can write non-autocommit approach by next two days, if its needed. Have a nice day! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1419) Solr bolt should handle tick tuples
[ https://issues.apache.org/jira/browse/STORM-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088769#comment-15088769 ] ASF GitHub Bot commented on STORM-1419: --- Github user hmcl commented on the pull request: https://github.com/apache/storm/pull/977#issuecomment-169908195 +1 once the last few, minor, comments are addressed. @vesense Thanks for the great work! > Solr bolt should handle tick tuples > --- > > Key: STORM-1419 > URL: https://issues.apache.org/jira/browse/STORM-1419 > Project: Apache Storm > Issue Type: Bug > Components: storm-solr >Reporter: Xin Wang >Assignee: Xin Wang > > Solr bolt should handle tick tuples. > Forcing solr client commit when bolt received tick tuple. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1419) Solr bolt should handle tick tuples
[
https://issues.apache.org/jira/browse/STORM-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088767#comment-15088767
]
ASF GitHub Bot commented on STORM-1419:
---
Github user hmcl commented on a diff in the pull request:
https://github.com/apache/storm/pull/977#discussion_r49161066
--- Diff:
external/storm-solr/src/main/java/org/apache/storm/solr/config/SolrConfig.java
---
@@ -39,4 +40,13 @@ public SolrConfig(String zkHostString) {
public String getZkHostString() {
return zkHostString;
}
+
+public int getTickTupleInterval() {
+return tickTupleInterval;
+}
+
+public void setTickTupleInterval(int tickTupleInterval) {
--- End diff --
@vesense Can you please delete this set method and pass the
tickTupleInterval in the constructor, or use a builder pattern. This object
should be immutable and fully initialized once passed to the Bolt.
> Solr bolt should handle tick tuples
> ---
>
> Key: STORM-1419
> URL: https://issues.apache.org/jira/browse/STORM-1419
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-solr
>Reporter: Xin Wang
>Assignee: Xin Wang
>
> Solr bolt should handle tick tuples.
> Forcing solr client commit when bolt received tick tuple.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: [STORM-1419] Solr bolt should handle tick tupl...
Github user hmcl commented on the pull request: https://github.com/apache/storm/pull/977#issuecomment-169908195 +1 once the last few, minor, comments are addressed. @vesense Thanks for the great work! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-1419] Solr bolt should handle tick tupl...
Github user hmcl commented on a diff in the pull request:
https://github.com/apache/storm/pull/977#discussion_r49161066
--- Diff:
external/storm-solr/src/main/java/org/apache/storm/solr/config/SolrConfig.java
---
@@ -39,4 +40,13 @@ public SolrConfig(String zkHostString) {
public String getZkHostString() {
return zkHostString;
}
+
+public int getTickTupleInterval() {
+return tickTupleInterval;
+}
+
+public void setTickTupleInterval(int tickTupleInterval) {
--- End diff --
@vesense Can you please delete this set method and pass the
tickTupleInterval in the constructor, or use a builder pattern. This object
should be immutable and fully initialized once passed to the Bolt.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-1419) Solr bolt should handle tick tuples
[
https://issues.apache.org/jira/browse/STORM-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088765#comment-15088765
]
ASF GitHub Bot commented on STORM-1419:
---
Github user hmcl commented on a diff in the pull request:
https://github.com/apache/storm/pull/977#discussion_r49160962
--- Diff:
external/storm-solr/src/main/java/org/apache/storm/solr/bolt/SolrUpdateBolt.java
---
@@ -59,15 +63,24 @@ public SolrUpdateBolt(SolrConfig solrConfig, SolrMapper
solrMapper, SolrCommitSt
this.solrConfig = solrConfig;
this.solrMapper = solrMapper;
this.commitStgy = commitStgy;
-logger.debug("Created {} with the following configuration: " +
+LOG.debug("Created {} with the following configuration: " +
"[SolrConfig = {}], [SolrMapper = {}], [CommitStgy =
{}]",
this.getClass().getSimpleName(), solrConfig,
solrMapper, commitStgy);
}
+@Override
public void prepare(Map stormConf, TopologyContext context,
OutputCollector collector) {
this.collector = collector;
this.solrClient = new
CloudSolrClient(solrConfig.getZkHostString());
this.toCommitTuples = new ArrayList<>(capacity());
+this.tickTupleInterval = solrConfig.getTickTupleInterval();
+
+//set default tickTupleInterval
+if (stormConf.containsKey("topology.message.timeout.secs") &&
tickTupleInterval == 0) {
--- End diff --
@vesense Please create a private method called something
setDefaultTickTupleInterval() with this code and call it here.
> Solr bolt should handle tick tuples
> ---
>
> Key: STORM-1419
> URL: https://issues.apache.org/jira/browse/STORM-1419
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-solr
>Reporter: Xin Wang
>Assignee: Xin Wang
>
> Solr bolt should handle tick tuples.
> Forcing solr client commit when bolt received tick tuple.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: [STORM-1419] Solr bolt should handle tick tupl...
Github user hmcl commented on a diff in the pull request:
https://github.com/apache/storm/pull/977#discussion_r49160962
--- Diff:
external/storm-solr/src/main/java/org/apache/storm/solr/bolt/SolrUpdateBolt.java
---
@@ -59,15 +63,24 @@ public SolrUpdateBolt(SolrConfig solrConfig, SolrMapper
solrMapper, SolrCommitSt
this.solrConfig = solrConfig;
this.solrMapper = solrMapper;
this.commitStgy = commitStgy;
-logger.debug("Created {} with the following configuration: " +
+LOG.debug("Created {} with the following configuration: " +
"[SolrConfig = {}], [SolrMapper = {}], [CommitStgy =
{}]",
this.getClass().getSimpleName(), solrConfig,
solrMapper, commitStgy);
}
+@Override
public void prepare(Map stormConf, TopologyContext context,
OutputCollector collector) {
this.collector = collector;
this.solrClient = new
CloudSolrClient(solrConfig.getZkHostString());
this.toCommitTuples = new ArrayList<>(capacity());
+this.tickTupleInterval = solrConfig.getTickTupleInterval();
+
+//set default tickTupleInterval
+if (stormConf.containsKey("topology.message.timeout.secs") &&
tickTupleInterval == 0) {
--- End diff --
@vesense Please create a private method called something
setDefaultTickTupleInterval() with this code and call it here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-1419) Solr bolt should handle tick tuples
[
https://issues.apache.org/jira/browse/STORM-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088764#comment-15088764
]
ASF GitHub Bot commented on STORM-1419:
---
Github user hmcl commented on a diff in the pull request:
https://github.com/apache/storm/pull/977#discussion_r49160913
--- Diff:
external/storm-solr/src/main/java/org/apache/storm/solr/bolt/SolrUpdateBolt.java
---
@@ -131,6 +150,21 @@ private void failQueuedTuples(List
failedTuples) {
return queuedTuples;
}
+@Override
+public Map getComponentConfiguration() {
--- End diff --
@vesense I noticed that the body of this method is exactly the same for
AbstractHdfsBolt. Furthermore, it's very likely that many Storm connector bolts
will use the same code. Can you please write a helper method in TupleUtils and
call that method here and AbstractHdfsBolt.
> Solr bolt should handle tick tuples
> ---
>
> Key: STORM-1419
> URL: https://issues.apache.org/jira/browse/STORM-1419
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-solr
>Reporter: Xin Wang
>Assignee: Xin Wang
>
> Solr bolt should handle tick tuples.
> Forcing solr client commit when bolt received tick tuple.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: [STORM-1419] Solr bolt should handle tick tupl...
Github user hmcl commented on a diff in the pull request:
https://github.com/apache/storm/pull/977#discussion_r49160913
--- Diff:
external/storm-solr/src/main/java/org/apache/storm/solr/bolt/SolrUpdateBolt.java
---
@@ -131,6 +150,21 @@ private void failQueuedTuples(List
failedTuples) {
return queuedTuples;
}
+@Override
+public Map getComponentConfiguration() {
--- End diff --
@vesense I noticed that the body of this method is exactly the same for
AbstractHdfsBolt. Furthermore, it's very likely that many Storm connector bolts
will use the same code. Can you please write a helper method in TupleUtils and
call that method here and AbstractHdfsBolt.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-1406) MQTT Support
[
https://issues.apache.org/jira/browse/STORM-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088761#comment-15088761
]
ASF GitHub Bot commented on STORM-1406:
---
Github user ptgoetz commented on a diff in the pull request:
https://github.com/apache/storm/pull/991#discussion_r49160828
--- Diff:
external/storm-mqtt/core/src/main/java/org/apache/storm/mqtt/trident/MqttPublishFunction.java
---
@@ -0,0 +1,84 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.mqtt.trident;
+
+import backtype.storm.Config;
+import backtype.storm.task.OutputCollector;
+import backtype.storm.topology.FailedException;
+import org.apache.storm.mqtt.MqttMessage;
+import org.apache.storm.mqtt.common.MqttOptions;
+import org.apache.storm.mqtt.MqttTupleMapper;
+import org.apache.storm.mqtt.common.MqttPublisher;
+import org.apache.storm.mqtt.common.SslUtils;
+import org.apache.storm.mqtt.ssl.KeyStoreLoader;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import storm.trident.operation.BaseFunction;
+import storm.trident.operation.TridentCollector;
+import storm.trident.operation.TridentOperationContext;
+import storm.trident.tuple.TridentTuple;
+
+import java.util.Map;
+
+public class MqttPublishFunction extends BaseFunction {
--- End diff --
I thought about that distinction. Ultimately, I decided upon a function
based on the following criteria:
1, There is no bulk publishing mechanism with MQTT, so there wouldn't be
any performance benefit in processing tuples in batch.
2. Aside from publishing a message with a QoS of 2, we don't have any way
to truly determine if it reached all subscribers, only the broker, and if the
broker dies...??
3. Exactly once is virtually impossible with MQTT in a distributed
environment.
4. Most MQTT use cases that need to scale avoid QoS 2 because of the
overhead.
That being said, I'm perfectly open to exploring a MQTT State
implementation if it would be useful.
> MQTT Support
>
>
> Key: STORM-1406
> URL: https://issues.apache.org/jira/browse/STORM-1406
> Project: Apache Storm
> Issue Type: Improvement
>Reporter: P. Taylor Goetz
>Assignee: P. Taylor Goetz
>
> MQTT is a lightweight publish/subscribe protocol frequently used in IoT
> applications.
> Further information can be found at http://mqtt.org
> Initial features include:
> * Full MQTT support (e.g. last will, QoS 0-2, retain, etc.)
> * Spout implementation(s) for subscribing to MQTT topics
> * A bolt implementation for publishing MQTT messages
> * A trident function implementation for publishing MQTT messages
> * Authentication and TLS/SSL support
> * User-defined "mappers" for converting MQTT messages to tuples (subscribers)
> * User-defined "mappers" for converting tuples to MQTT messages (publishers)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1419) Solr bolt should handle tick tuples
[
https://issues.apache.org/jira/browse/STORM-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088760#comment-15088760
]
ASF GitHub Bot commented on STORM-1419:
---
Github user hmcl commented on a diff in the pull request:
https://github.com/apache/storm/pull/977#discussion_r49160816
--- Diff:
external/storm-solr/src/main/java/org/apache/storm/solr/bolt/SolrUpdateBolt.java
---
@@ -92,9 +108,12 @@ private void ack(Tuple tuple) throws
SolrServerException, IOException {
if (commitStgy == null) {
collector.ack(tuple);
} else {
-toCommitTuples.add(tuple);
-commitStgy.update();
-if (commitStgy.commit()) {
+final boolean isTickTuple = TupleUtils.isTick(tuple);
+if (!isTickTuple) {
--- End diff --
Can you please add the inline comment: // Don't ack tick tuples.
> Solr bolt should handle tick tuples
> ---
>
> Key: STORM-1419
> URL: https://issues.apache.org/jira/browse/STORM-1419
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-solr
>Reporter: Xin Wang
>Assignee: Xin Wang
>
> Solr bolt should handle tick tuples.
> Forcing solr client commit when bolt received tick tuple.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-1406: Add MQTT Support
Github user ptgoetz commented on a diff in the pull request:
https://github.com/apache/storm/pull/991#discussion_r49160828
--- Diff:
external/storm-mqtt/core/src/main/java/org/apache/storm/mqtt/trident/MqttPublishFunction.java
---
@@ -0,0 +1,84 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.mqtt.trident;
+
+import backtype.storm.Config;
+import backtype.storm.task.OutputCollector;
+import backtype.storm.topology.FailedException;
+import org.apache.storm.mqtt.MqttMessage;
+import org.apache.storm.mqtt.common.MqttOptions;
+import org.apache.storm.mqtt.MqttTupleMapper;
+import org.apache.storm.mqtt.common.MqttPublisher;
+import org.apache.storm.mqtt.common.SslUtils;
+import org.apache.storm.mqtt.ssl.KeyStoreLoader;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import storm.trident.operation.BaseFunction;
+import storm.trident.operation.TridentCollector;
+import storm.trident.operation.TridentOperationContext;
+import storm.trident.tuple.TridentTuple;
+
+import java.util.Map;
+
+public class MqttPublishFunction extends BaseFunction {
--- End diff --
I thought about that distinction. Ultimately, I decided upon a function
based on the following criteria:
1, There is no bulk publishing mechanism with MQTT, so there wouldn't be
any performance benefit in processing tuples in batch.
2. Aside from publishing a message with a QoS of 2, we don't have any way
to truly determine if it reached all subscribers, only the broker, and if the
broker dies...??
3. Exactly once is virtually impossible with MQTT in a distributed
environment.
4. Most MQTT use cases that need to scale avoid QoS 2 because of the
overhead.
That being said, I'm perfectly open to exploring a MQTT State
implementation if it would be useful.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-1419] Solr bolt should handle tick tupl...
Github user hmcl commented on a diff in the pull request:
https://github.com/apache/storm/pull/977#discussion_r49160816
--- Diff:
external/storm-solr/src/main/java/org/apache/storm/solr/bolt/SolrUpdateBolt.java
---
@@ -92,9 +108,12 @@ private void ack(Tuple tuple) throws
SolrServerException, IOException {
if (commitStgy == null) {
collector.ack(tuple);
} else {
-toCommitTuples.add(tuple);
-commitStgy.update();
-if (commitStgy.commit()) {
+final boolean isTickTuple = TupleUtils.isTick(tuple);
+if (!isTickTuple) {
--- End diff --
Can you please add the inline comment: // Don't ack tick tuples.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-1452) Worker "profiler" actions broken by default
[ https://issues.apache.org/jira/browse/STORM-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088753#comment-15088753 ] Derek Dagit commented on STORM-1452: Here is a [partial patch|https://gist.github.com/d2r/ab15f312a5af1d9525af] that makes everything work, but we still need to disable the jfr feature by default. > Worker "profiler" actions broken by default > --- > > Key: STORM-1452 > URL: https://issues.apache.org/jira/browse/STORM-1452 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 0.11.0 >Reporter: Derek Dagit >Priority: Blocker > > * The profiler script flight.bash is not packaged by default. > * The default options enable the Oracle specific flight-recorder that > requires a support subscription. > The option to enable the profiler should not be enabled by default. Other > actions such as worker restart, debugging, and heap can remain enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1430] ui worker checkboxes
GitHub user d2r opened a pull request: https://github.com/apache/storm/pull/995 [STORM-1430] ui worker checkboxes This adds new checkboxes to the Executors tables of the component pages. Selecting these will enable the debugging/profiling actions to be performed on multiple workers while making it much easier to choose workersâespecially for topologies with hundreds of workers. Look at the documentation to see what has changed. This is a little hard to test, because of [STORM-1452](https://issues.apache.org/jira/browse/STORM-1452). As a work-around, manually copy flight.bash to somewhere on the supervisor process's PATH and it should work. This also cleans up some tooltip placement and removes some extra bindings from core.clj, and I put these in their own commits so it is clearer. You can merge this pull request into a Git repository by running: $ git pull https://github.com/d2r/storm storm-1430-ui-worker-checkboxes Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/995.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #995 commit 181a46dfa561ba863a170efc5b688e5427ee7089 Author: Derek Dagit Date: 2016-01-08T05:06:03Z removes unnecessary bindings commit a1c3ebbc249e5d35ae59531b424f90b930471378 Author: Derek Dagit Date: 2016-01-08T05:22:40Z corrects tooltip placement commit 497aa0991b05b1e8296511bbc3aacf0d9a0830ad Author: Derek Dagit Date: 2016-01-08T05:26:17Z Adds debugging checkboxes and links to exec table commit c34266924a0bcb7c17bddc0a602580e892153651 Author: Derek Dagit Date: 2016-01-08T05:27:30Z updates profiling docs for action checkboxes & links --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1430) UI Worker Functions: Replace pick-list with buttons
[ https://issues.apache.org/jira/browse/STORM-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088747#comment-15088747 ] ASF GitHub Bot commented on STORM-1430: --- GitHub user d2r opened a pull request: https://github.com/apache/storm/pull/995 [STORM-1430] ui worker checkboxes This adds new checkboxes to the Executors tables of the component pages. Selecting these will enable the debugging/profiling actions to be performed on multiple workers while making it much easier to choose workers—especially for topologies with hundreds of workers. Look at the documentation to see what has changed. This is a little hard to test, because of [STORM-1452](https://issues.apache.org/jira/browse/STORM-1452). As a work-around, manually copy flight.bash to somewhere on the supervisor process's PATH and it should work. This also cleans up some tooltip placement and removes some extra bindings from core.clj, and I put these in their own commits so it is clearer. You can merge this pull request into a Git repository by running: $ git pull https://github.com/d2r/storm storm-1430-ui-worker-checkboxes Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/995.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #995 commit 181a46dfa561ba863a170efc5b688e5427ee7089 Author: Derek Dagit Date: 2016-01-08T05:06:03Z removes unnecessary bindings commit a1c3ebbc249e5d35ae59531b424f90b930471378 Author: Derek Dagit Date: 2016-01-08T05:22:40Z corrects tooltip placement commit 497aa0991b05b1e8296511bbc3aacf0d9a0830ad Author: Derek Dagit Date: 2016-01-08T05:26:17Z Adds debugging checkboxes and links to exec table commit c34266924a0bcb7c17bddc0a602580e892153651 Author: Derek Dagit Date: 2016-01-08T05:27:30Z updates profiling docs for action checkboxes & links > UI Worker Functions: Replace pick-list with buttons > --- > > Key: STORM-1430 > URL: https://issues.apache.org/jira/browse/STORM-1430 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Affects Versions: 0.11.0 >Reporter: Derek Dagit >Assignee: Derek Dagit >Priority: Minor > > The Component Page in the UI has a very useful set of functions that allow us > to profile a worker, to restart a worker, to create a stack trace, or to save > the heap. > When there are many workers, finding the correct worker host:port from the > pick list is tedious. > As a user, I would like the worker functions presented as buttons directly in > the Executors summary tables, so that I can easily request the functions when > there are lots of workers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1199) Create HDFS Spout
[ https://issues.apache.org/jira/browse/STORM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088721#comment-15088721 ] ASF GitHub Bot commented on STORM-1199: --- Github user arunmahadevan commented on the pull request: https://github.com/apache/storm/pull/936#issuecomment-169898710 +1 Please squash the commits and add some comments/docs on the locking semantics https://github.com/apache/storm/pull/936#discussion_r49158800 > Create HDFS Spout > - > > Key: STORM-1199 > URL: https://issues.apache.org/jira/browse/STORM-1199 > Project: Apache Storm > Issue Type: New Feature >Reporter: Roshan Naik >Assignee: Roshan Naik > Attachments: HDFSSpoutforStorm v2.pdf, HDFSSpoutforStorm.pdf, > hdfs-spout.1.patch > > > Create an HDFS spout so that Storm can suck in data from files in a HDFS > directory -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1199 : HDFS Spout Functionally complete....
Github user arunmahadevan commented on the pull request: https://github.com/apache/storm/pull/936#issuecomment-169898710 +1 Please squash the commits and add some comments/docs on the locking semantics https://github.com/apache/storm/pull/936#discussion_r49158800 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1199) Create HDFS Spout
[
https://issues.apache.org/jira/browse/STORM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088717#comment-15088717
]
ASF GitHub Bot commented on STORM-1199:
---
Github user arunmahadevan commented on a diff in the pull request:
https://github.com/apache/storm/pull/936#discussion_r49158800
--- Diff:
external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/DirLock.java ---
@@ -0,0 +1,133 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.hdfs.spout;
+
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.DistributedFileSystem;
+import org.apache.storm.hdfs.common.HdfsUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+
+/**
+ * Facility to synchronize access to HDFS directory. The lock itself is
represented
+ * as a file in the same directory. Relies on atomic file creation.
+ */
+public class DirLock {
+ private FileSystem fs;
+ private final Path lockFile;
+ public static final String DIR_LOCK_FILE = "DIRLOCK";
+ private static final Logger LOG = LoggerFactory.getLogger(DirLock.class);
+ private DirLock(FileSystem fs, Path lockFile) throws IOException {
+if( fs.isDirectory(lockFile) ) {
+ throw new IllegalArgumentException(lockFile.toString() + " is not a
directory");
+}
+this.fs = fs;
+this.lockFile = lockFile;
+ }
+
+ /** Get a lock on file if not already locked
+ *
+ * @param fs
+ * @param dir the dir on which to get a lock
+ * @return The lock object if it the lock was acquired. Returns null if
the dir is already locked.
+ * @throws IOException if there were errors
+ */
+ public static DirLock tryLock(FileSystem fs, Path dir) throws
IOException {
+Path lockFile = getDirLockFile(dir);
+
+try {
+ FSDataOutputStream ostream = HdfsUtils.tryCreateFile(fs, lockFile);
--- End diff --
The RawLocalFileSystem implementation does not seem to take care of
atomicity -
[link](https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java#L299).
In the distributed mode name node could guarantee atomicity by locking, but I
havent checked the details. Did you run the tests in local or distributed mode
? Anyways it would be hard to get the order and timing right for the tests to
fail.
It might be good to add some comments/docs to call out that the locking
semantics depends on the underlying implementation guarantees.
> Create HDFS Spout
> -
>
> Key: STORM-1199
> URL: https://issues.apache.org/jira/browse/STORM-1199
> Project: Apache Storm
> Issue Type: New Feature
>Reporter: Roshan Naik
>Assignee: Roshan Naik
> Attachments: HDFSSpoutforStorm v2.pdf, HDFSSpoutforStorm.pdf,
> hdfs-spout.1.patch
>
>
> Create an HDFS spout so that Storm can suck in data from files in a HDFS
> directory
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-1199 : HDFS Spout Functionally complete....
Github user arunmahadevan commented on a diff in the pull request:
https://github.com/apache/storm/pull/936#discussion_r49158800
--- Diff:
external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/DirLock.java ---
@@ -0,0 +1,133 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.hdfs.spout;
+
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.DistributedFileSystem;
+import org.apache.storm.hdfs.common.HdfsUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+
+/**
+ * Facility to synchronize access to HDFS directory. The lock itself is
represented
+ * as a file in the same directory. Relies on atomic file creation.
+ */
+public class DirLock {
+ private FileSystem fs;
+ private final Path lockFile;
+ public static final String DIR_LOCK_FILE = "DIRLOCK";
+ private static final Logger LOG = LoggerFactory.getLogger(DirLock.class);
+ private DirLock(FileSystem fs, Path lockFile) throws IOException {
+if( fs.isDirectory(lockFile) ) {
+ throw new IllegalArgumentException(lockFile.toString() + " is not a
directory");
+}
+this.fs = fs;
+this.lockFile = lockFile;
+ }
+
+ /** Get a lock on file if not already locked
+ *
+ * @param fs
+ * @param dir the dir on which to get a lock
+ * @return The lock object if it the lock was acquired. Returns null if
the dir is already locked.
+ * @throws IOException if there were errors
+ */
+ public static DirLock tryLock(FileSystem fs, Path dir) throws
IOException {
+Path lockFile = getDirLockFile(dir);
+
+try {
+ FSDataOutputStream ostream = HdfsUtils.tryCreateFile(fs, lockFile);
--- End diff --
The RawLocalFileSystem implementation does not seem to take care of
atomicity -
[link](https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java#L299).
In the distributed mode name node could guarantee atomicity by locking, but I
havent checked the details. Did you run the tests in local or distributed mode
? Anyways it would be hard to get the order and timing right for the tests to
fail.
It might be good to add some comments/docs to call out that the locking
semantics depends on the underlying implementation guarantees.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-1419) Solr bolt should handle tick tuples
[ https://issues.apache.org/jira/browse/STORM-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088712#comment-15088712 ] ASF GitHub Bot commented on STORM-1419: --- Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/977#issuecomment-169896957 +1 > Solr bolt should handle tick tuples > --- > > Key: STORM-1419 > URL: https://issues.apache.org/jira/browse/STORM-1419 > Project: Apache Storm > Issue Type: Bug > Components: storm-solr >Reporter: Xin Wang >Assignee: Xin Wang > > Solr bolt should handle tick tuples. > Forcing solr client commit when bolt received tick tuple. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1419] Solr bolt should handle tick tupl...
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/977#issuecomment-169896957 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1419) Solr bolt should handle tick tuples
[ https://issues.apache.org/jira/browse/STORM-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088696#comment-15088696 ] ASF GitHub Bot commented on STORM-1419: --- Github user vesense commented on the pull request: https://github.com/apache/storm/pull/977#issuecomment-169894466 @hmcl comments addressed. > Solr bolt should handle tick tuples > --- > > Key: STORM-1419 > URL: https://issues.apache.org/jira/browse/STORM-1419 > Project: Apache Storm > Issue Type: Bug > Components: storm-solr >Reporter: Xin Wang >Assignee: Xin Wang > > Solr bolt should handle tick tuples. > Forcing solr client commit when bolt received tick tuple. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1419] Solr bolt should handle tick tupl...
Github user vesense commented on the pull request: https://github.com/apache/storm/pull/977#issuecomment-169894466 @hmcl comments addressed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1419) Solr bolt should handle tick tuples
[
https://issues.apache.org/jira/browse/STORM-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088687#comment-15088687
]
ASF GitHub Bot commented on STORM-1419:
---
Github user vesense commented on a diff in the pull request:
https://github.com/apache/storm/pull/977#discussion_r49157386
--- Diff:
external/storm-solr/src/main/java/org/apache/storm/solr/bolt/SolrUpdateBolt.java
---
@@ -59,16 +64,24 @@ public SolrUpdateBolt(SolrConfig solrConfig, SolrMapper
solrMapper, SolrCommitSt
this.solrConfig = solrConfig;
this.solrMapper = solrMapper;
this.commitStgy = commitStgy;
-logger.debug("Created {} with the following configuration: " +
+LOG.debug("Created {} with the following configuration: " +
"[SolrConfig = {}], [SolrMapper = {}], [CommitStgy =
{}]",
this.getClass().getSimpleName(), solrConfig,
solrMapper, commitStgy);
}
+@Override
public void prepare(Map stormConf, TopologyContext context,
OutputCollector collector) {
this.collector = collector;
this.solrClient = new
CloudSolrClient(solrConfig.getZkHostString());
this.toCommitTuples = new ArrayList<>(capacity());
+//set default tickTupleInterval if interval is zero
+if (stormConf.containsKey("topology.message.timeout.secs") &&
tickTupleInterval == 0) {
--- End diff --
I prefer to keep this code in `prepare()` method since no `stormConf` in
SolrConfig.
> Solr bolt should handle tick tuples
> ---
>
> Key: STORM-1419
> URL: https://issues.apache.org/jira/browse/STORM-1419
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-solr
>Reporter: Xin Wang
>Assignee: Xin Wang
>
> Solr bolt should handle tick tuples.
> Forcing solr client commit when bolt received tick tuple.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: [STORM-1419] Solr bolt should handle tick tupl...
Github user vesense commented on a diff in the pull request:
https://github.com/apache/storm/pull/977#discussion_r49157386
--- Diff:
external/storm-solr/src/main/java/org/apache/storm/solr/bolt/SolrUpdateBolt.java
---
@@ -59,16 +64,24 @@ public SolrUpdateBolt(SolrConfig solrConfig, SolrMapper
solrMapper, SolrCommitSt
this.solrConfig = solrConfig;
this.solrMapper = solrMapper;
this.commitStgy = commitStgy;
-logger.debug("Created {} with the following configuration: " +
+LOG.debug("Created {} with the following configuration: " +
"[SolrConfig = {}], [SolrMapper = {}], [CommitStgy =
{}]",
this.getClass().getSimpleName(), solrConfig,
solrMapper, commitStgy);
}
+@Override
public void prepare(Map stormConf, TopologyContext context,
OutputCollector collector) {
this.collector = collector;
this.solrClient = new
CloudSolrClient(solrConfig.getZkHostString());
this.toCommitTuples = new ArrayList<>(capacity());
+//set default tickTupleInterval if interval is zero
+if (stormConf.containsKey("topology.message.timeout.secs") &&
tickTupleInterval == 0) {
--- End diff --
I prefer to keep this code in `prepare()` method since no `stormConf` in
SolrConfig.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-1419) Solr bolt should handle tick tuples
[
https://issues.apache.org/jira/browse/STORM-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088678#comment-15088678
]
ASF GitHub Bot commented on STORM-1419:
---
Github user vesense commented on a diff in the pull request:
https://github.com/apache/storm/pull/977#discussion_r49157203
--- Diff:
external/storm-solr/src/main/java/org/apache/storm/solr/bolt/SolrUpdateBolt.java
---
@@ -92,9 +108,17 @@ private void ack(Tuple tuple) throws
SolrServerException, IOException {
if (commitStgy == null) {
collector.ack(tuple);
} else {
-toCommitTuples.add(tuple);
-commitStgy.update();
-if (commitStgy.commit()) {
+boolean forceCommit = false;
+if (TupleUtils.isTick(tuple)) {
+LOG.debug("TICK! forcing solr client commit");
+collector.ack(tuple);
+forceCommit = true;
+} else {
+toCommitTuples.add(tuple);
+commitStgy.update();
+}
+
+if (forceCommit || commitStgy.commit()) {
--- End diff --
OK
> Solr bolt should handle tick tuples
> ---
>
> Key: STORM-1419
> URL: https://issues.apache.org/jira/browse/STORM-1419
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-solr
>Reporter: Xin Wang
>Assignee: Xin Wang
>
> Solr bolt should handle tick tuples.
> Forcing solr client commit when bolt received tick tuple.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1419) Solr bolt should handle tick tuples
[
https://issues.apache.org/jira/browse/STORM-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088679#comment-15088679
]
ASF GitHub Bot commented on STORM-1419:
---
Github user vesense commented on a diff in the pull request:
https://github.com/apache/storm/pull/977#discussion_r49157210
--- Diff:
external/storm-solr/src/main/java/org/apache/storm/solr/bolt/SolrUpdateBolt.java
---
@@ -92,9 +108,17 @@ private void ack(Tuple tuple) throws
SolrServerException, IOException {
if (commitStgy == null) {
collector.ack(tuple);
} else {
-toCommitTuples.add(tuple);
-commitStgy.update();
-if (commitStgy.commit()) {
+boolean forceCommit = false;
+if (TupleUtils.isTick(tuple)) {
+LOG.debug("TICK! forcing solr client commit");
+collector.ack(tuple);
--- End diff --
OK
> Solr bolt should handle tick tuples
> ---
>
> Key: STORM-1419
> URL: https://issues.apache.org/jira/browse/STORM-1419
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-solr
>Reporter: Xin Wang
>Assignee: Xin Wang
>
> Solr bolt should handle tick tuples.
> Forcing solr client commit when bolt received tick tuple.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1419) Solr bolt should handle tick tuples
[
https://issues.apache.org/jira/browse/STORM-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088680#comment-15088680
]
ASF GitHub Bot commented on STORM-1419:
---
Github user vesense commented on a diff in the pull request:
https://github.com/apache/storm/pull/977#discussion_r49157240
--- Diff:
external/storm-solr/src/main/java/org/apache/storm/solr/bolt/SolrUpdateBolt.java
---
@@ -131,6 +155,26 @@ private void failQueuedTuples(List
failedTuples) {
return queuedTuples;
}
+public SolrUpdateBolt withTickIntervalSecs(int tickTupleInterval) {
--- End diff --
OK. Thx.
> Solr bolt should handle tick tuples
> ---
>
> Key: STORM-1419
> URL: https://issues.apache.org/jira/browse/STORM-1419
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-solr
>Reporter: Xin Wang
>Assignee: Xin Wang
>
> Solr bolt should handle tick tuples.
> Forcing solr client commit when bolt received tick tuple.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: [STORM-1419] Solr bolt should handle tick tupl...
Github user vesense commented on a diff in the pull request:
https://github.com/apache/storm/pull/977#discussion_r49157240
--- Diff:
external/storm-solr/src/main/java/org/apache/storm/solr/bolt/SolrUpdateBolt.java
---
@@ -131,6 +155,26 @@ private void failQueuedTuples(List
failedTuples) {
return queuedTuples;
}
+public SolrUpdateBolt withTickIntervalSecs(int tickTupleInterval) {
--- End diff --
OK. Thx.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-1419] Solr bolt should handle tick tupl...
Github user vesense commented on a diff in the pull request:
https://github.com/apache/storm/pull/977#discussion_r49157203
--- Diff:
external/storm-solr/src/main/java/org/apache/storm/solr/bolt/SolrUpdateBolt.java
---
@@ -92,9 +108,17 @@ private void ack(Tuple tuple) throws
SolrServerException, IOException {
if (commitStgy == null) {
collector.ack(tuple);
} else {
-toCommitTuples.add(tuple);
-commitStgy.update();
-if (commitStgy.commit()) {
+boolean forceCommit = false;
+if (TupleUtils.isTick(tuple)) {
+LOG.debug("TICK! forcing solr client commit");
+collector.ack(tuple);
+forceCommit = true;
+} else {
+toCommitTuples.add(tuple);
+commitStgy.update();
+}
+
+if (forceCommit || commitStgy.commit()) {
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-1419] Solr bolt should handle tick tupl...
Github user vesense commented on a diff in the pull request:
https://github.com/apache/storm/pull/977#discussion_r49157210
--- Diff:
external/storm-solr/src/main/java/org/apache/storm/solr/bolt/SolrUpdateBolt.java
---
@@ -92,9 +108,17 @@ private void ack(Tuple tuple) throws
SolrServerException, IOException {
if (commitStgy == null) {
collector.ack(tuple);
} else {
-toCommitTuples.add(tuple);
-commitStgy.update();
-if (commitStgy.commit()) {
+boolean forceCommit = false;
+if (TupleUtils.isTick(tuple)) {
+LOG.debug("TICK! forcing solr client commit");
+collector.ack(tuple);
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-1199 : HDFS Spout Functionally complete....
Github user roshannaik commented on the pull request: https://github.com/apache/storm/pull/936#issuecomment-169882606 @arunmahadevan have also addressed your comment about releasing the lock files on exception with the latest push. Good catch(..) ! ;-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1199) Create HDFS Spout
[ https://issues.apache.org/jira/browse/STORM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088642#comment-15088642 ] ASF GitHub Bot commented on STORM-1199: --- Github user roshannaik commented on the pull request: https://github.com/apache/storm/pull/936#issuecomment-169882606 @arunmahadevan have also addressed your comment about releasing the lock files on exception with the latest push. Good catch(..) ! ;-) > Create HDFS Spout > - > > Key: STORM-1199 > URL: https://issues.apache.org/jira/browse/STORM-1199 > Project: Apache Storm > Issue Type: New Feature >Reporter: Roshan Naik >Assignee: Roshan Naik > Attachments: HDFSSpoutforStorm v2.pdf, HDFSSpoutforStorm.pdf, > hdfs-spout.1.patch > > > Create an HDFS spout so that Storm can suck in data from files in a HDFS > directory -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1421) Couldn't extract resources in the supervisor
[ https://issues.apache.org/jira/browse/STORM-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088552#comment-15088552 ] beginner commented on STORM-1421: - can you help ? > Couldn't extract resources in the supervisor > - > > Key: STORM-1421 > URL: https://issues.apache.org/jira/browse/STORM-1421 > Project: Apache Storm > Issue Type: Bug >Reporter: beginner >Assignee: beginner >Priority: Critical > > i found in the supervisor log file that couldn't extract resources from > /supervisor/tmp/7c3f723c-6b7f-4919-a01c-5a87b872dda7/stormjar.jar > then got still hasn't start > please tell me which document or information i can share it to help in fixing > i searched more to fix it but couldn't and some people told me that it's a > bug in storm > i hope i can help here to fix it fastly beacuse i'm on this error months and > my project stopped until i fix this error > i'm using storm-0.8.2 > i ran this topology in local mode sucessfully but when i tried to submit it i > got above error > i'm trying now to submit it in real cluster > so i have 2 laptops one of them 1 "server" and other 2 i installed storm and > zookeeper in both laptop and installed ssh in server laptop > i started to connect 2 from server and start zookeeper then ran nimbus in > server 1 but supervisor as i read i ran it in both but still error and i > tried to ran supervisor in 2 only got same error > this is my configuration of zookeeper in both lap > tickTime=2000 > dataDir=/var/zookeeper > clientPort=2181 > initLimit=5 > syncLimit=2 > and this is for storm.yaml > storm.zookeeper.servers: > - "ip address if laptop 2 " > nimbus.host: "ip address of laptop 1" > storm.zookeeper.port: 2181 > storm.local.dir: "/var/storms" > java.library.path: "/usr/lib/jvm/java-6-openjdk-amd64:/usr/local/lib" > nimbus.thrift.port: 6627 > supervisor.childopts: "-Djava.net.preferIPv4Stack=true" > nimbus.childopts: "-Xmx1024m -Djava.net.preferIPv4Stack=true" > worker.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true" > ui.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true" > supervisor.slots.ports: > > - 6700 > - 6701 > - 6702 > - 6703 > /etc/hosts > # 127.0.0.1 localhost > my ip address user-Lenovo > #127.0.1.1localhost2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1452) Worker "profiler" actions broken by default
[ https://issues.apache.org/jira/browse/STORM-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088515#comment-15088515 ] John Fang commented on STORM-1452: -- +1 > Worker "profiler" actions broken by default > --- > > Key: STORM-1452 > URL: https://issues.apache.org/jira/browse/STORM-1452 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 0.11.0 >Reporter: Derek Dagit >Priority: Blocker > > * The profiler script flight.bash is not packaged by default. > * The default options enable the Oracle specific flight-recorder that > requires a support subscription. > The option to enable the profiler should not be enabled by default. Other > actions such as worker restart, debugging, and heap can remain enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1452) Worker "profiler" actions broken by default
[ https://issues.apache.org/jira/browse/STORM-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088486#comment-15088486 ] Derek Dagit commented on STORM-1452: Created as a Blocker since the presentation is very visible on the UI page. It should not be too hard to fix. > Worker "profiler" actions broken by default > --- > > Key: STORM-1452 > URL: https://issues.apache.org/jira/browse/STORM-1452 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 0.11.0 >Reporter: Derek Dagit >Priority: Blocker > > * The profiler script flight.bash is not packaged by default. > * The default options enable the Oracle specific flight-recorder that > requires a support subscription. > The option to enable the profiler should not be enabled by default. Other > actions such as worker restart, debugging, and heap can remain enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-1452) Worker "profiler" actions broken by default
Derek Dagit created STORM-1452: -- Summary: Worker "profiler" actions broken by default Key: STORM-1452 URL: https://issues.apache.org/jira/browse/STORM-1452 Project: Apache Storm Issue Type: Bug Components: storm-core Affects Versions: 0.11.0 Reporter: Derek Dagit Priority: Blocker * The profiler script flight.bash is not packaged by default. * The default options enable the Oracle specific flight-recorder that requires a support subscription. The option to enable the profiler should not be enabled by default. Other actions such as worker restart, debugging, and heap can remain enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (STORM-1430) UI Worker Functions: Replace pick-list with buttons
[ https://issues.apache.org/jira/browse/STORM-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Derek Dagit reassigned STORM-1430: -- Assignee: Derek Dagit > UI Worker Functions: Replace pick-list with buttons > --- > > Key: STORM-1430 > URL: https://issues.apache.org/jira/browse/STORM-1430 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Affects Versions: 0.11.0 >Reporter: Derek Dagit >Assignee: Derek Dagit >Priority: Minor > > The Component Page in the UI has a very useful set of functions that allow us > to profile a worker, to restart a worker, to create a stack trace, or to save > the heap. > When there are many workers, finding the correct worker host:port from the > pick list is tedious. > As a user, I would like the worker functions presented as buttons directly in > the Executors summary tables, so that I can easily request the functions when > there are lots of workers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1419) Solr bolt should handle tick tuples
[
https://issues.apache.org/jira/browse/STORM-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088321#comment-15088321
]
ASF GitHub Bot commented on STORM-1419:
---
Github user hmcl commented on a diff in the pull request:
https://github.com/apache/storm/pull/977#discussion_r49138053
--- Diff:
external/storm-solr/src/main/java/org/apache/storm/solr/bolt/SolrUpdateBolt.java
---
@@ -59,16 +64,24 @@ public SolrUpdateBolt(SolrConfig solrConfig, SolrMapper
solrMapper, SolrCommitSt
this.solrConfig = solrConfig;
this.solrMapper = solrMapper;
this.commitStgy = commitStgy;
-logger.debug("Created {} with the following configuration: " +
+LOG.debug("Created {} with the following configuration: " +
"[SolrConfig = {}], [SolrMapper = {}], [CommitStgy =
{}]",
this.getClass().getSimpleName(), solrConfig,
solrMapper, commitStgy);
}
+@Override
public void prepare(Map stormConf, TopologyContext context,
OutputCollector collector) {
this.collector = collector;
this.solrClient = new
CloudSolrClient(solrConfig.getZkHostString());
this.toCommitTuples = new ArrayList<>(capacity());
+//set default tickTupleInterval if interval is zero
+if (stormConf.containsKey("topology.message.timeout.secs") &&
tickTupleInterval == 0) {
--- End diff --
@vesense This code should go in the SolrConfig object
> Solr bolt should handle tick tuples
> ---
>
> Key: STORM-1419
> URL: https://issues.apache.org/jira/browse/STORM-1419
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-solr
>Reporter: Xin Wang
>Assignee: Xin Wang
>
> Solr bolt should handle tick tuples.
> Forcing solr client commit when bolt received tick tuple.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: [STORM-1419] Solr bolt should handle tick tupl...
Github user hmcl commented on a diff in the pull request:
https://github.com/apache/storm/pull/977#discussion_r49138053
--- Diff:
external/storm-solr/src/main/java/org/apache/storm/solr/bolt/SolrUpdateBolt.java
---
@@ -59,16 +64,24 @@ public SolrUpdateBolt(SolrConfig solrConfig, SolrMapper
solrMapper, SolrCommitSt
this.solrConfig = solrConfig;
this.solrMapper = solrMapper;
this.commitStgy = commitStgy;
-logger.debug("Created {} with the following configuration: " +
+LOG.debug("Created {} with the following configuration: " +
"[SolrConfig = {}], [SolrMapper = {}], [CommitStgy =
{}]",
this.getClass().getSimpleName(), solrConfig,
solrMapper, commitStgy);
}
+@Override
public void prepare(Map stormConf, TopologyContext context,
OutputCollector collector) {
this.collector = collector;
this.solrClient = new
CloudSolrClient(solrConfig.getZkHostString());
this.toCommitTuples = new ArrayList<>(capacity());
+//set default tickTupleInterval if interval is zero
+if (stormConf.containsKey("topology.message.timeout.secs") &&
tickTupleInterval == 0) {
--- End diff --
@vesense This code should go in the SolrConfig object
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-1499] Fix Kafka spout to maintain backw...
Github user zhuoliu commented on the pull request: https://github.com/apache/storm/pull/994#issuecomment-169832464 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1419) Solr bolt should handle tick tuples
[
https://issues.apache.org/jira/browse/STORM-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088297#comment-15088297
]
ASF GitHub Bot commented on STORM-1419:
---
Github user hmcl commented on a diff in the pull request:
https://github.com/apache/storm/pull/977#discussion_r49136662
--- Diff:
external/storm-solr/src/main/java/org/apache/storm/solr/bolt/SolrUpdateBolt.java
---
@@ -131,6 +155,26 @@ private void failQueuedTuples(List
failedTuples) {
return queuedTuples;
}
+public SolrUpdateBolt withTickIntervalSecs(int tickTupleInterval) {
--- End diff --
@vesense this violates encapsulation and should be avoided
> Solr bolt should handle tick tuples
> ---
>
> Key: STORM-1419
> URL: https://issues.apache.org/jira/browse/STORM-1419
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-solr
>Reporter: Xin Wang
>Assignee: Xin Wang
>
> Solr bolt should handle tick tuples.
> Forcing solr client commit when bolt received tick tuple.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: [STORM-1419] Solr bolt should handle tick tupl...
Github user hmcl commented on a diff in the pull request:
https://github.com/apache/storm/pull/977#discussion_r49136662
--- Diff:
external/storm-solr/src/main/java/org/apache/storm/solr/bolt/SolrUpdateBolt.java
---
@@ -131,6 +155,26 @@ private void failQueuedTuples(List
failedTuples) {
return queuedTuples;
}
+public SolrUpdateBolt withTickIntervalSecs(int tickTupleInterval) {
--- End diff --
@vesense this violates encapsulation and should be avoided
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-1419] Solr bolt should handle tick tupl...
Github user hmcl commented on a diff in the pull request:
https://github.com/apache/storm/pull/977#discussion_r49136481
--- Diff:
external/storm-solr/src/main/java/org/apache/storm/solr/bolt/SolrUpdateBolt.java
---
@@ -92,9 +108,17 @@ private void ack(Tuple tuple) throws
SolrServerException, IOException {
if (commitStgy == null) {
collector.ack(tuple);
} else {
-toCommitTuples.add(tuple);
-commitStgy.update();
-if (commitStgy.commit()) {
+boolean forceCommit = false;
+if (TupleUtils.isTick(tuple)) {
+LOG.debug("TICK! forcing solr client commit");
+collector.ack(tuple);
--- End diff --
tick tuples don't need to be acked
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-1419] Solr bolt should handle tick tupl...
Github user hmcl commented on a diff in the pull request:
https://github.com/apache/storm/pull/977#discussion_r49136454
--- Diff:
external/storm-solr/src/main/java/org/apache/storm/solr/bolt/SolrUpdateBolt.java
---
@@ -92,9 +108,17 @@ private void ack(Tuple tuple) throws
SolrServerException, IOException {
if (commitStgy == null) {
collector.ack(tuple);
} else {
-toCommitTuples.add(tuple);
-commitStgy.update();
-if (commitStgy.commit()) {
+boolean forceCommit = false;
+if (TupleUtils.isTick(tuple)) {
+LOG.debug("TICK! forcing solr client commit");
+collector.ack(tuple);
+forceCommit = true;
+} else {
+toCommitTuples.add(tuple);
+commitStgy.update();
+}
+
+if (forceCommit || commitStgy.commit()) {
--- End diff --
@vesense Can you please follow te suggestion that I gave as the code is
much cleaner.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-1419) Solr bolt should handle tick tuples
[
https://issues.apache.org/jira/browse/STORM-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088292#comment-15088292
]
ASF GitHub Bot commented on STORM-1419:
---
Github user hmcl commented on a diff in the pull request:
https://github.com/apache/storm/pull/977#discussion_r49136481
--- Diff:
external/storm-solr/src/main/java/org/apache/storm/solr/bolt/SolrUpdateBolt.java
---
@@ -92,9 +108,17 @@ private void ack(Tuple tuple) throws
SolrServerException, IOException {
if (commitStgy == null) {
collector.ack(tuple);
} else {
-toCommitTuples.add(tuple);
-commitStgy.update();
-if (commitStgy.commit()) {
+boolean forceCommit = false;
+if (TupleUtils.isTick(tuple)) {
+LOG.debug("TICK! forcing solr client commit");
+collector.ack(tuple);
--- End diff --
tick tuples don't need to be acked
> Solr bolt should handle tick tuples
> ---
>
> Key: STORM-1419
> URL: https://issues.apache.org/jira/browse/STORM-1419
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-solr
>Reporter: Xin Wang
>Assignee: Xin Wang
>
> Solr bolt should handle tick tuples.
> Forcing solr client commit when bolt received tick tuple.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1419) Solr bolt should handle tick tuples
[
https://issues.apache.org/jira/browse/STORM-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088291#comment-15088291
]
ASF GitHub Bot commented on STORM-1419:
---
Github user hmcl commented on a diff in the pull request:
https://github.com/apache/storm/pull/977#discussion_r49136454
--- Diff:
external/storm-solr/src/main/java/org/apache/storm/solr/bolt/SolrUpdateBolt.java
---
@@ -92,9 +108,17 @@ private void ack(Tuple tuple) throws
SolrServerException, IOException {
if (commitStgy == null) {
collector.ack(tuple);
} else {
-toCommitTuples.add(tuple);
-commitStgy.update();
-if (commitStgy.commit()) {
+boolean forceCommit = false;
+if (TupleUtils.isTick(tuple)) {
+LOG.debug("TICK! forcing solr client commit");
+collector.ack(tuple);
+forceCommit = true;
+} else {
+toCommitTuples.add(tuple);
+commitStgy.update();
+}
+
+if (forceCommit || commitStgy.commit()) {
--- End diff --
@vesense Can you please follow te suggestion that I gave as the code is
much cleaner.
> Solr bolt should handle tick tuples
> ---
>
> Key: STORM-1419
> URL: https://issues.apache.org/jira/browse/STORM-1419
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-solr
>Reporter: Xin Wang
>Assignee: Xin Wang
>
> Solr bolt should handle tick tuples.
> Forcing solr client commit when bolt received tick tuple.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1420) solr CountBasedCommit should reset count=0 after commited
[ https://issues.apache.org/jira/browse/STORM-1420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088278#comment-15088278 ] ASF GitHub Bot commented on STORM-1420: --- Github user hmcl commented on the pull request: https://github.com/apache/storm/pull/978#issuecomment-169828477 @vesense why is this change necessary? > solr CountBasedCommit should reset count=0 after commited > - > > Key: STORM-1420 > URL: https://issues.apache.org/jira/browse/STORM-1420 > Project: Apache Storm > Issue Type: Bug > Components: storm-solr >Reporter: Xin Wang >Assignee: Xin Wang > > The variable count should reset to 0, if not so it will increase infinitely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1420] solr CountBasedCommit should rese...
Github user hmcl commented on the pull request: https://github.com/apache/storm/pull/978#issuecomment-169828477 @vesense why is this change necessary? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1199) Create HDFS Spout
[
https://issues.apache.org/jira/browse/STORM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088135#comment-15088135
]
ASF GitHub Bot commented on STORM-1199:
---
Github user roshannaik commented on a diff in the pull request:
https://github.com/apache/storm/pull/936#discussion_r49127649
--- Diff:
external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/DirLock.java ---
@@ -0,0 +1,133 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.hdfs.spout;
+
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.DistributedFileSystem;
+import org.apache.storm.hdfs.common.HdfsUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+
+/**
+ * Facility to synchronize access to HDFS directory. The lock itself is
represented
+ * as a file in the same directory. Relies on atomic file creation.
+ */
+public class DirLock {
+ private FileSystem fs;
+ private final Path lockFile;
+ public static final String DIR_LOCK_FILE = "DIRLOCK";
+ private static final Logger LOG = LoggerFactory.getLogger(DirLock.class);
+ private DirLock(FileSystem fs, Path lockFile) throws IOException {
+if( fs.isDirectory(lockFile) ) {
+ throw new IllegalArgumentException(lockFile.toString() + " is not a
directory");
+}
+this.fs = fs;
+this.lockFile = lockFile;
+ }
+
+ /** Get a lock on file if not already locked
+ *
+ * @param fs
+ * @param dir the dir on which to get a lock
+ * @return The lock object if it the lock was acquired. Returns null if
the dir is already locked.
+ * @throws IOException if there were errors
+ */
+ public static DirLock tryLock(FileSystem fs, Path dir) throws
IOException {
+Path lockFile = getDirLockFile(dir);
+
+try {
+ FSDataOutputStream ostream = HdfsUtils.tryCreateFile(fs, lockFile);
--- End diff --
I believe that is for other file systems (such as AWS) that are supported
via that class. I double checked on on HDFS semantics by talking to some HDFS
committers. Have also added UTs to check for the requisite Hdfs atomicity/other
semantics related to create, append and delete as an added safeguard. We will
see failures in these tests if the guarantees are being violated. These are the
relevant UT tests: TestHdfsSemantics.* TestFileLock.testConcurrentLocking,
TestDirLock.testConcurrentLocking
> Create HDFS Spout
> -
>
> Key: STORM-1199
> URL: https://issues.apache.org/jira/browse/STORM-1199
> Project: Apache Storm
> Issue Type: New Feature
>Reporter: Roshan Naik
>Assignee: Roshan Naik
> Attachments: HDFSSpoutforStorm v2.pdf, HDFSSpoutforStorm.pdf,
> hdfs-spout.1.patch
>
>
> Create an HDFS spout so that Storm can suck in data from files in a HDFS
> directory
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1199) Create HDFS Spout
[ https://issues.apache.org/jira/browse/STORM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088143#comment-15088143 ] ASF GitHub Bot commented on STORM-1199: --- Github user roshannaik commented on a diff in the pull request: https://github.com/apache/storm/pull/936#discussion_r49128132 --- Diff: external/storm-hdfs/README.md --- @@ -405,7 +410,123 @@ On worker hosts the bolt/trident-state code will use the keytab file with princi Namenode. This method is little dangerous as you need to ensure all workers have the keytab file at the same location and you need to remember this as you bring up new hosts in the cluster. -## License +--- + +# HDFS Spout + +Hdfs spout is intended to allow feeding data into Storm from a HDFS directory. +It will actively monitor the directory to consume any new files that appear in the directory. +HDFS spout does not support Trident currently. + +**Impt**: Hdfs spout assumes that the files being made visible to it in the monitored directory +are NOT actively being written to. Only after a file is completely written should it be made +visible to the spout. This can be achieved by either writing the files out to another directory +and once completely written, move it to the monitored directory. Alternatively the file +can be created with a '.ignore' suffix in the monitored directory and after data is completely +written, rename it without the suffix. File names with a '.ignore' suffix are ignored +by the spout. + +When the spout is actively consuming a file, it renames the file with a '.inprogress' suffix. +After consuming all the contents in the file, the file will be moved to a configurable *done* +directory and the '.inprogress' suffix will be dropped. + +**Concurrency** If multiple spout instances are used in the topology, each instance will consume +a different file. Synchronization among spout instances is done using lock files created in a +(by default) '.lock' subdirectory under the monitored directory. A file with the same name +as the file being consumed (without the in progress suffix) is created in the lock directory. +Once the file is completely consumed, the corresponding lock file is deleted. + +**Recovery from failure** +Periodically, the spout also records progress information wrt to how much of the file has been +consumed in the lock file. In case of an crash of the spout instance (or force kill of topology) +another spout can take over the file and resume from the location recorded in the lock file. + +Certain error conditions (such spout crashing) can leave behind lock files without deleting them. +Such a stale lock file also indicates that the corresponding input file has also not been completely +processed. When detected, ownership of such stale lock files will be transferred to another spout. +The configuration 'hdfsspout.lock.timeout.sec' is used to specify the duration of inactivity after +which lock files should be considered stale. For lock file ownership transfer to succeed, the HDFS +lease on the file (from prev lock owner) should have expired. Spouts scan for stale lock files +before selecting the next file for consumption. + +**Lock on *.lock* Directory** +Hdfs spout instances create a *DIRLOCK* file in the .lock directory to co-ordinate certain accesses to +the .lock dir itself. A spout will try to create it when it needs access to the .lock directory and +then delete it when done. In case of a topology crash or force kill, this file may not get deleted. --- End diff -- you are right.. it should. I should reword it. Thanks! > Create HDFS Spout > - > > Key: STORM-1199 > URL: https://issues.apache.org/jira/browse/STORM-1199 > Project: Apache Storm > Issue Type: New Feature >Reporter: Roshan Naik >Assignee: Roshan Naik > Attachments: HDFSSpoutforStorm v2.pdf, HDFSSpoutforStorm.pdf, > hdfs-spout.1.patch > > > Create an HDFS spout so that Storm can suck in data from files in a HDFS > directory -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1199 : HDFS Spout Functionally complete....
Github user roshannaik commented on a diff in the pull request: https://github.com/apache/storm/pull/936#discussion_r49128132 --- Diff: external/storm-hdfs/README.md --- @@ -405,7 +410,123 @@ On worker hosts the bolt/trident-state code will use the keytab file with princi Namenode. This method is little dangerous as you need to ensure all workers have the keytab file at the same location and you need to remember this as you bring up new hosts in the cluster. -## License +--- + +# HDFS Spout + +Hdfs spout is intended to allow feeding data into Storm from a HDFS directory. +It will actively monitor the directory to consume any new files that appear in the directory. +HDFS spout does not support Trident currently. + +**Impt**: Hdfs spout assumes that the files being made visible to it in the monitored directory +are NOT actively being written to. Only after a file is completely written should it be made +visible to the spout. This can be achieved by either writing the files out to another directory +and once completely written, move it to the monitored directory. Alternatively the file +can be created with a '.ignore' suffix in the monitored directory and after data is completely +written, rename it without the suffix. File names with a '.ignore' suffix are ignored +by the spout. + +When the spout is actively consuming a file, it renames the file with a '.inprogress' suffix. +After consuming all the contents in the file, the file will be moved to a configurable *done* +directory and the '.inprogress' suffix will be dropped. + +**Concurrency** If multiple spout instances are used in the topology, each instance will consume +a different file. Synchronization among spout instances is done using lock files created in a +(by default) '.lock' subdirectory under the monitored directory. A file with the same name +as the file being consumed (without the in progress suffix) is created in the lock directory. +Once the file is completely consumed, the corresponding lock file is deleted. + +**Recovery from failure** +Periodically, the spout also records progress information wrt to how much of the file has been +consumed in the lock file. In case of an crash of the spout instance (or force kill of topology) +another spout can take over the file and resume from the location recorded in the lock file. + +Certain error conditions (such spout crashing) can leave behind lock files without deleting them. +Such a stale lock file also indicates that the corresponding input file has also not been completely +processed. When detected, ownership of such stale lock files will be transferred to another spout. +The configuration 'hdfsspout.lock.timeout.sec' is used to specify the duration of inactivity after +which lock files should be considered stale. For lock file ownership transfer to succeed, the HDFS +lease on the file (from prev lock owner) should have expired. Spouts scan for stale lock files +before selecting the next file for consumption. + +**Lock on *.lock* Directory** +Hdfs spout instances create a *DIRLOCK* file in the .lock directory to co-ordinate certain accesses to +the .lock dir itself. A spout will try to create it when it needs access to the .lock directory and +then delete it when done. In case of a topology crash or force kill, this file may not get deleted. --- End diff -- you are right.. it should. I should reword it. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-1199 : HDFS Spout Functionally complete....
Github user roshannaik commented on a diff in the pull request:
https://github.com/apache/storm/pull/936#discussion_r49127649
--- Diff:
external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/DirLock.java ---
@@ -0,0 +1,133 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.hdfs.spout;
+
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.DistributedFileSystem;
+import org.apache.storm.hdfs.common.HdfsUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+
+/**
+ * Facility to synchronize access to HDFS directory. The lock itself is
represented
+ * as a file in the same directory. Relies on atomic file creation.
+ */
+public class DirLock {
+ private FileSystem fs;
+ private final Path lockFile;
+ public static final String DIR_LOCK_FILE = "DIRLOCK";
+ private static final Logger LOG = LoggerFactory.getLogger(DirLock.class);
+ private DirLock(FileSystem fs, Path lockFile) throws IOException {
+if( fs.isDirectory(lockFile) ) {
+ throw new IllegalArgumentException(lockFile.toString() + " is not a
directory");
+}
+this.fs = fs;
+this.lockFile = lockFile;
+ }
+
+ /** Get a lock on file if not already locked
+ *
+ * @param fs
+ * @param dir the dir on which to get a lock
+ * @return The lock object if it the lock was acquired. Returns null if
the dir is already locked.
+ * @throws IOException if there were errors
+ */
+ public static DirLock tryLock(FileSystem fs, Path dir) throws
IOException {
+Path lockFile = getDirLockFile(dir);
+
+try {
+ FSDataOutputStream ostream = HdfsUtils.tryCreateFile(fs, lockFile);
--- End diff --
I believe that is for other file systems (such as AWS) that are supported
via that class. I double checked on on HDFS semantics by talking to some HDFS
committers. Have also added UTs to check for the requisite Hdfs atomicity/other
semantics related to create, append and delete as an added safeguard. We will
see failures in these tests if the guarantees are being violated. These are the
relevant UT tests: TestHdfsSemantics.* TestFileLock.testConcurrentLocking,
TestDirLock.testConcurrentLocking
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Created] (STORM-1451) Storm topology submission can take upto 5 minutes in HA mode when zookeeper reconnects. Nimbus discovery can fail when zookeeper reconnect happens.
Priyank Shah created STORM-1451: --- Summary: Storm topology submission can take upto 5 minutes in HA mode when zookeeper reconnects. Nimbus discovery can fail when zookeeper reconnect happens. Key: STORM-1451 URL: https://issues.apache.org/jira/browse/STORM-1451 Project: Apache Storm Issue Type: Bug Components: storm-core Reporter: Priyank Shah We discovered couple of issues when testing storm under vagrant clusters. 1. When a nimbus zookeeper connection is dropped and reconnected the ephemeral entry for that host under /nimbuses gets deleted and is not auto recreated when reconnection happens. This means even though nimbus is up no client will be able to actually discover it. To address this issue we now have a listener that listens for RECONNECT events and recreates the entry. 2. Zookeeper is eventual consistent when multiple clients are involved. In practice we did not notice this issue but in the vagrant cluster due to resource constrained it was pretty evident that updates created by leader nimbuses were not observed by other nimbus host unless they waited for a few second. Due to this topology submission can take upto 5 minutes which is super bad user experience. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Potential license issue for storm-cassandra
Okay, I just got all the storm-cassandra tests to pass without cassandra-unit by writing a quick-and-dirty embedded C* instance. I’ve got some cleanup and refactoring to do, but it looks like a viable alternative. I should have a pull request up soon. -Taylor > On Jan 7, 2016, at 2:55 PM, Bobby Evans wrote: > > That seems reasonable to me. Thanks for looking into this Taylor. > - Bobby > >On Wednesday, January 6, 2016 4:28 PM, P. Taylor Goetz > wrote: > > > More information: > > The fact that the code is in our repo is not a problem, it just means that we > we can’t release until that is rectified by removing the LGPL dependency. > When Storm first entered incubation we had a dependency on 0mq (LGPL), which > didn’t stop the code from being imported into the Apache repo, but it had to > be removed before we could officially release. > > Looking at the unit tests in question, it doesn’t seem like it would be hard > to migrate away from cassandra-unit. I’ve used cassandra-unit in the past (a > while ago), and eventually migrated away from it by spinning up an in-memory > C* instance and populating it with the necessary data from within the unit > tests, which is what cassandra-unit seems to be used for here. > > Florian — would you be able to help migrate off of cassandra-unit, possibly > considering the approach I mentioned? If not I may be able to find some time > to do it. > > If for some reason we can’t migrate by the time we’re ready to release, we > can just delete the tests/dependency. But I’d at least try to > preserve/migrate them since I believe they have value. > > -Taylor > > >> On Jan 6, 2016, at 4:13 PM, P. Taylor Goetz wrote: >> >> Let me do some quick research before you rip anything out. >> >> -Taylor >> >>> On Jan 6, 2016, at 4:10 PM, Bobby Evans wrote: >>> >>> Yes lets just remove them for now, and then file a follow up JIRA to add >>> back in tests. >>> - Bobby >>> >>> On Wednesday, January 6, 2016 3:01 PM, Jungtaek Lim >>> wrote: >>> >>> >>> Sorry missed link, https://issues.apache.org/jira/browse/STORM-1445 >>> >>> 2016년 1월 7일 (목) 오전 5:59, Jungtaek Lim 님이 작성: >>> Filed STORM-1445. Seems like unit tests are completely relying on cassandra-unit. I don't have experience with Cassandra, so I couldn't convert current tests to not use cassandra-unit. If we think we're fine to remove whole unit tests for storm-cassandra, I'll remove it and submit pull request right now. If we still need unit tests for storm-cassandra, I'd love to let sponsors of storm-cassandra module takes care of it. (Maybe we can file a new issue which handles new unit tests.) Best, Jungtaek Lim (HeartSaVioR) 2016년 1월 7일 (목) 오전 5:31, Bobby Evans 님이 작성: > Yes pull it out for now, and we may have to talk to someone in legal at > apache if there is something else we need to do. We have not done a > release with storm-cassandra yet, so we are probably safe. Because of > that > please file a JIRA and put up a pull request like normal. > - Bobby > > On Wednesday, January 6, 2016 7:35 AM, Jungtaek Lim < > kabh...@gmail.com> wrote: > > > Hi devs, > > Digging into the test failures on storm-cassandra, I saw license of > cassandra-unit is LGPL v3 by chance. > > https://github.com/jsevellec/cassandra-unit > > From http://www.apache.org/legal/resolved.html, the page describes that > > LGPL-licensed works must not be included in Apache Products > > > but I don't know much details on license so clarification would be much > appreciated. > > Should we get rid of cassandra-unit? I sought the alternatives, but > nothing > found. > > Best, > Jungtaek Lim (HeartSaVioR) > > -- > Name : Jungtaek Lim > Blog : http://medium.com/@heartsavior > Twitter : http://twitter.com/heartsavior > LinkedIn : http://www.linkedin.com/in/heartsavior > > > >>> >> > > signature.asc Description: Message signed with OpenPGP using GPGMail
[GitHub] storm pull request: STORM-1202: Migrate APIs to org.apache.storm, ...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/889#issuecomment-169780775 I just filed http://dev.clojure.org/jira/browse/CLJ-1876 for the issue I was seeing with the require. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1189) client.getClusterInfo() fails with Required field 'nimbus_uptime_secs' is unset!
[ https://issues.apache.org/jira/browse/STORM-1189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15087910#comment-15087910 ] ASF GitHub Bot commented on STORM-1189: --- Github user YuChem commented on the pull request: https://github.com/apache/storm/pull/868#issuecomment-169778882 @spicoflorin: I have encountered the same issue. Did you get it resolved? Would you mind get some tips on how to workaround? > client.getClusterInfo() fails with Required field 'nimbus_uptime_secs' is > unset! > > > Key: STORM-1189 > URL: https://issues.apache.org/jira/browse/STORM-1189 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 0.10.0 > Environment: CentOS 7, Hortonworks 2.3, Java API >Reporter: Michael Sonst >Assignee: Robert Joseph Evans > Fix For: 0.11.0 > > > Hi, > we are about to upgrade our cluster from 0.9.1-incubating to 0.10.0. I'm > currently testing whether the functionality is still working with 0.10.0 in > VMs. > On 0.10.0 the following throws the exception mentioned below: > ... > NimbusClient nimbusClient = NimbusClient.getConfiguredClient(readConfig); > Client client = nimbusClient.getClient(); > ClusterSummary clusterInfo = client.getClusterInfo(); > Exception: > org.apache.thrift7.protocol.TProtocolException: Required field > 'nimbus_uptime_secs' is unset! > Struct:ClusterSummary(supervisors:[SupervisorSummary(host:n3.t3k.siemens.com, > uptime_secs:24297, num_workers:4, num_used_workers:0, > supervisor_id:4a87f03b-ea91-4b12-8cb9-9f7e26703b26, > version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[]) > at > backtype.storm.generated.ClusterSummary.validate(ClusterSummary.java:515) > at > backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.read(ClusterSummary.java:613) > at > backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.read(ClusterSummary.java:549) > at backtype.storm.generated.ClusterSummary.read(ClusterSummary.java:473) > at > backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_resultStandardScheme.read(Nimbus.java:16546) > at > backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_resultStandardScheme.read(Nimbus.java:16531) > at > backtype.storm.generated.Nimbus$getClusterInfo_result.read(Nimbus.java:16470) > at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:78) > at > backtype.storm.generated.Nimbus$Client.recv_getClusterInfo(Nimbus.java:569) > at > backtype.storm.generated.Nimbus$Client.getClusterInfo(Nimbus.java:557) > Switching back to 0.9.1-incubating showed, that the code still works fine > there. > BR Michael -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1189: Maintain wire compatability with 0...
Github user YuChem commented on the pull request: https://github.com/apache/storm/pull/868#issuecomment-169778882 @spicoflorin: I have encountered the same issue. Did you get it resolved? Would you mind get some tips on how to workaround? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-468) java.io.NotSerializableException should be explained
[ https://issues.apache.org/jira/browse/STORM-468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15087901#comment-15087901 ] ASF GitHub Bot commented on STORM-468: -- Github user dossett commented on the pull request: https://github.com/apache/storm/pull/477#issuecomment-169778180 @jangie Yes, my comment was based a sloppy reading of your PR. Nice work! > java.io.NotSerializableException should be explained > > > Key: STORM-468 > URL: https://issues.apache.org/jira/browse/STORM-468 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 0.9.2-incubating > Environment: Any >Reporter: Jason Kania >Priority: Minor > Labels: newbie > > The occurrence of the NotSerializableException and how to avoid it should be > better documented and the error from the code should be expanded to include > some human readable message because it took a lot of searching to find out > that the spouts and bolts need to create their variables in the prepare > method in order to avoid this problem. > The error text output could state that this is how to solve the problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1202: Migrate APIs to org.apache.storm, ...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/889#issuecomment-169775907 OK I finally tracked down some intermittent failures I was seeing when running the clojure word count example. It looks like there is an issue in clojure where how we are doing the require does not seem to be thread safe. We should be good to go now if @d2r you want to take a final look. The previous failure is not related and is one of the random storm-kafka failures. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1202) Migrate APIs to org.apache.storm, but try to provide backwards compatability as a bridge
[ https://issues.apache.org/jira/browse/STORM-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15087556#comment-15087556 ] ASF GitHub Bot commented on STORM-1202: --- Github user d2r commented on the pull request: https://github.com/apache/storm/pull/889#issuecomment-169690819 Looks OK to me overall. It would be nice to have a follow-on Jira Issue or Sub-Task to remove/revert the changes we need to bootstrap the testing for this (dev-tools, etc.). > I have tested this by running a 0.10.0-SNAPSHOT storm starter topologies against a cluster running this patch (along with the move_package.sh executed). We found that with word_count, the .clj source file had been removed from the transformed jar, but we expected that this .clj would be present, but transformed. > Migrate APIs to org.apache.storm, but try to provide backwards compatability > as a bridge > > > Key: STORM-1202 > URL: https://issues.apache.org/jira/browse/STORM-1202 > Project: Apache Storm > Issue Type: New Feature >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > > We want to move the storm classpaths to org.apache.storm wherever possible, > but we also want to provide backwards compatibility for user facing APIs > whenever possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1202: Migrate APIs to org.apache.storm, ...
Github user d2r commented on the pull request: https://github.com/apache/storm/pull/889#issuecomment-169690819 Looks OK to me overall. It would be nice to have a follow-on Jira Issue or Sub-Task to remove/revert the changes we need to bootstrap the testing for this (dev-tools, etc.). > I have tested this by running a 0.10.0-SNAPSHOT storm starter topologies against a cluster running this patch (along with the move_package.sh executed). We found that with word_count, the .clj source file had been removed from the transformed jar, but we expected that this .clj would be present, but transformed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1202) Migrate APIs to org.apache.storm, but try to provide backwards compatability as a bridge
[
https://issues.apache.org/jira/browse/STORM-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15087506#comment-15087506
]
ASF GitHub Bot commented on STORM-1202:
---
Github user d2r commented on a diff in the pull request:
https://github.com/apache/storm/pull/889#discussion_r49078953
--- Diff: storm-rename-hack/pom.xml ---
@@ -0,0 +1,107 @@
+
+
+http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ 4.0.0
+
+ storm
+ org.apache.storm
+ 0.11.0-SNAPSHOT
+ ../pom.xml
+
+
+ org.apache.storm
+ storm-rename-hack
+ jar
+
+ storm-rename-hack
+
+
+3.0
+5.0.2
+
+
+
+
+ org.apache.storm
+ storm-core
+ ${project.version}
+ provided
+
+
+ com.google.guava
+ guava
+
+
+ org.ow2.asm
+ asm
+ ${asmVersion}
+
+
+ org.ow2.asm
+ asm-commons
+ ${asmVersion}
+
+
--- End diff --
nit: check indentation
> Migrate APIs to org.apache.storm, but try to provide backwards compatability
> as a bridge
>
>
> Key: STORM-1202
> URL: https://issues.apache.org/jira/browse/STORM-1202
> Project: Apache Storm
> Issue Type: New Feature
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>
> We want to move the storm classpaths to org.apache.storm wherever possible,
> but we also want to provide backwards compatibility for user facing APIs
> whenever possible.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-1202: Migrate APIs to org.apache.storm, ...
Github user d2r commented on a diff in the pull request:
https://github.com/apache/storm/pull/889#discussion_r49078953
--- Diff: storm-rename-hack/pom.xml ---
@@ -0,0 +1,107 @@
+
+
+http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ 4.0.0
+
+ storm
+ org.apache.storm
+ 0.11.0-SNAPSHOT
+ ../pom.xml
+
+
+ org.apache.storm
+ storm-rename-hack
+ jar
+
+ storm-rename-hack
+
+
+3.0
+5.0.2
+
+
+
+
+ org.apache.storm
+ storm-core
+ ${project.version}
+ provided
+
+
+ com.google.guava
+ guava
+
+
+ org.ow2.asm
+ asm
+ ${asmVersion}
+
+
+ org.ow2.asm
+ asm-commons
+ ${asmVersion}
+
+
--- End diff --
nit: check indentation
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-1499] Fix Kafka spout to maintain backw...
GitHub user arunmahadevan opened a pull request: https://github.com/apache/storm/pull/994 [STORM-1499] Fix Kafka spout to maintain backward compatibility STORM-1220 introduced some changes where in one place it passes ByteBuffer as is instead of byte[]. Existing bolts (0.10.0) expects a byte[] and fails with "java.lang.RuntimeException: java.lang.ClassCastException: java.nio.HeapByteBuffer cannot be cast to [B " This patch addresses the issue by emiting byte[] instead of ByteBuffer. You can merge this pull request into a Git repository by running: $ git pull https://github.com/arunmahadevan/storm STORM-1499 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/994.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #994 commit 1eca8a3398679dce7675e72d91527ac05335bd54 Author: Arun Mahadevan Date: 2016-01-07T09:56:00Z [STORM-1499] Fix Kafka spout to maintain backward compatibility STORM-1220 introduced some changes where in one place it passes ByteBuffer as is instead of byte[]. Existing bolts (0.10.0) expects a byte[] and fails with "java.lang.RuntimeException: java.lang.ClassCastException: java.nio.HeapByteBuffer cannot be cast to [B " This patch addresses the issue by emiting byte[] instead of ByteBuffer. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
