[jira] [Commented] (STORM-766) Supervisor summary should include the version.
[ https://issues.apache.org/jira/browse/STORM-766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503690#comment-14503690 ] ASF GitHub Bot commented on STORM-766: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/529#issuecomment-94573682 I ran a simple manual test where I had an old supervisor and a new nimbus and everything worked well. I am +1 for merging this in once the merge conflict is resolved. Supervisor summary should include the version. -- Key: STORM-766 URL: https://issues.apache.org/jira/browse/STORM-766 Project: Apache Storm Issue Type: Bug Affects Versions: 0.10.0 Reporter: Parth Brahmbhatt Assignee: Sanket Chintapalli Priority: Minor Fix For: 0.10.0 With the support for rolling upgrade, different nodes in the cluster can run different versions of storm. We should include the version in SupervisorSummary just like NimbusSummary so admins can identify nodes that needs upgrading/downgrading from UI. As part of this change I will also add a supervisor/log link in the ui. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Upgrading Storm to use Netty 4.0 or higher
I would like to know what the benefits of upgrading would be. Upgrading a dependency, especially something as core as this, carries with it risk. Once we know the benefits we can weigh that against the risk. On Mon, Apr 20, 2015 at 1:54 PM, Bobby Evans ev...@yahoo-inc.com.invalid wrote: We have not really explored going to netty 4.0. I know at some point we will probably have to switch over to use it, but for the time being most of the dependencies that we have seen still use netty 3.X, but if you have code to use netty 4.x and want to contribute I would be happy to review it and see what we can do to support that. Even if it involves doing some shading to support it. - Bobby On Friday, April 17, 2015 10:48 PM, Julian Stephen julian.step...@gmail.com wrote: Hi All, I am quite curious to know if there is any interest or activity around upgrading Storm to use Netty 4.0 or higher. From what I understand, Storm (at least till the 0.10.0 dev branch I checked out) still depends on Netty 3.x. and Netty 4.x is not compatible with 3.x code Link http://netty.io/wiki/new-and-noteworthy-in-4.0.html. I could not find any related issues in the Apache Storm Jira, and is curious to know if anyone has explored implications and impact of such a change. Regards, Julian -- Twitter: @nathanmarz http://nathanmarz.com
[GitHub] storm pull request: JIRA STORM-766 (Include version info in the se...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/529#issuecomment-94573682 I ran a simple manual test where I had an old supervisor and a new nimbus and everything worked well. I am +1 for merging this in once the merge conflict is resolved. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-786: KafkaBolt should ack tick tuples
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/522#issuecomment-94575511 +1 looks good to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-786) KafkaBolt should ack tick tuples
[ https://issues.apache.org/jira/browse/STORM-786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503749#comment-14503749 ] Robert Joseph Evans commented on STORM-786: --- I merged this into master, [~miguno] thanks for the fix. KafkaBolt should ack tick tuples Key: STORM-786 URL: https://issues.apache.org/jira/browse/STORM-786 Project: Apache Storm Issue Type: Bug Components: storm-kafka Affects Versions: 0.11.0 Reporter: Michael Noll Assignee: Michael Noll Priority: Minor Fix For: 0.11.0 STORM-512 (KafkaBolt doesn't handle ticks properly) adds special-casing of tick tuples. What is missing in the patch is that the input tuple, when it is a tick tuple, should be properly acked like normal tuples. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-789) Enhance topology context sent to multi-lang bolts and spouts
[ https://issues.apache.org/jira/browse/STORM-789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503576#comment-14503576 ] ASF GitHub Bot commented on STORM-789: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/525#issuecomment-94564020 @dan-blanchard It doesn't. 0.11.0 is on master so that is where I have defaulted putting things. In general @ptgoetz has kind of been acting as the gate keeper for 0.10.x-branch, so if he thinks this should go in there I am happy to cherry-pick it back. Enhance topology context sent to multi-lang bolts and spouts Key: STORM-789 URL: https://issues.apache.org/jira/browse/STORM-789 Project: Apache Storm Issue Type: Improvement Reporter: Dan Blanchard Assignee: Dan Blanchard I'm about to submit a pull request that adds a lot more contextual information to the context JSON dictionary that gets sent to multi-lang bolts and spouts as part of their initial handshake. These additions will make is so non-JVM developers have access to the sources, targets, and field names for their bolts and spouts. We will add support for this in streamparse (one of the more popular Python libraries for Storm) as soon as this gets merged in. Here are the details of what I've added: - componentid: the component ID for this task, so you don't have to look it up in task-component by taskid - streams: a list of all of the streams for this task - stream-outputfields: a mapping from stream names to output fields for that stream - stream-target-grouping: a mapping from stream names to the targets for this task to the kind of grouping they use. - sources-grouping: a mapping from the component IDs of the sources to their grouping. Note on groupings: groupings are either represented as a string (e.g., SHUFFLE) or a list of strings for field groupings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-789) Enhance topology context sent to multi-lang bolts and spouts
[ https://issues.apache.org/jira/browse/STORM-789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503578#comment-14503578 ] ASF GitHub Bot commented on STORM-789: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/525#issuecomment-94564062 oh I am +1 on the current patch. Enhance topology context sent to multi-lang bolts and spouts Key: STORM-789 URL: https://issues.apache.org/jira/browse/STORM-789 Project: Apache Storm Issue Type: Improvement Reporter: Dan Blanchard Assignee: Dan Blanchard I'm about to submit a pull request that adds a lot more contextual information to the context JSON dictionary that gets sent to multi-lang bolts and spouts as part of their initial handshake. These additions will make is so non-JVM developers have access to the sources, targets, and field names for their bolts and spouts. We will add support for this in streamparse (one of the more popular Python libraries for Storm) as soon as this gets merged in. Here are the details of what I've added: - componentid: the component ID for this task, so you don't have to look it up in task-component by taskid - streams: a list of all of the streams for this task - stream-outputfields: a mapping from stream names to output fields for that stream - stream-target-grouping: a mapping from stream names to the targets for this task to the kind of grouping they use. - sources-grouping: a mapping from the component IDs of the sources to their grouping. Note on groupings: groupings are either represented as a string (e.g., SHUFFLE) or a list of strings for field groupings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-789] Send more topology context to Mult...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/525#issuecomment-94564020 @dan-blanchard It doesn't. 0.11.0 is on master so that is where I have defaulted putting things. In general @ptgoetz has kind of been acting as the gate keeper for 0.10.x-branch, so if he thinks this should go in there I am happy to cherry-pick it back. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: Question about Nimbus$Client
So it does. If you want to turn it into a pull request with the JIRA number in the title of the pull request I would love to pull it in. - Bobby On Monday, April 20, 2015 3:35 PM, Matthias J. Sax mj...@informatik.hu-berlin.de wrote: Hi, I had a quick look into it. JavaDoc can simply be added to the thrift code and is copied into generated Java and Python source code files. I modified storm.thrift and re-generated the code as described in DEVELOPER.md: cd storm-core/src sh genthrift.sh I pushed my changed to my git repo, in case you want to have a quick look: https://github.com/mjsax/storm I opened a JIRA, too: https://issues.apache.org/jira/browse/STORM-792 If you like it, I can open a pull request. Of course, it would be nice to add some more comments to all methods and interfaces. -Matthias On 04/20/2015 07:52 PM, Bobby Evans wrote: I totally agree there is a lot in the documentation that would be good to do. For the generated code I'm not totally sure how to make javadocs work. If you want to file a JIRA for this we can look at it. - Bobby On Saturday, April 18, 2015 9:49 AM, Matthias J. Sax mj...@informatik.hu-berlin.de wrote: Thanks! It is a petty, that this information in not documented via JavaDoc... That would make life much easier and would save time for everybody (= no need to ask and answer stupid questions on the mailing list) -Matthias On 04/17/2015 06:13 PM, Bobby Evans wrote: getTopology returns the compiled topology after nimbus has gotten its hands on it, so it has the ackers in it and the metrics consumers. getUserTopology returns the topology as the user submitted it. - Bobby On Friday, April 17, 2015 4:24 AM, Matthias J. Sax mj...@informatik.hu-berlin.de wrote: Dear all, the class backtype.storm.generated.Nimbus defines a nested class Nimbus$Cluster that offers the following two methods (both defined in Nimbus#Iface): public StormTopology getTopology(String id) throws NotAliveException, org.apache.thrift.TException; public StormTopology getUserTopology(String id) throws NotAliveException, org.apache.thrift.TException; What is the difference between both? I don't understand what the difference between a (regular?) topology and a user topology should be... From my understanding, there is only one type of topologies. Thanks for you help! -Matthias
[jira] [Commented] (STORM-786) KafkaBolt should ack tick tuples
[ https://issues.apache.org/jira/browse/STORM-786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503713#comment-14503713 ] ASF GitHub Bot commented on STORM-786: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/522#issuecomment-94575511 +1 looks good to me. KafkaBolt should ack tick tuples Key: STORM-786 URL: https://issues.apache.org/jira/browse/STORM-786 Project: Apache Storm Issue Type: Bug Components: storm-kafka Affects Versions: 0.11.0 Reporter: Michael Noll Priority: Minor STORM-512 (KafkaBolt doesn't handle ticks properly) adds special-casing of tick tuples. What is missing in the patch is that the input tuple, when it is a tick tuple, should be properly acked like normal tuples. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-786) KafkaBolt should ack tick tuples
[ https://issues.apache.org/jira/browse/STORM-786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated STORM-786: -- Assignee: Michael Noll KafkaBolt should ack tick tuples Key: STORM-786 URL: https://issues.apache.org/jira/browse/STORM-786 Project: Apache Storm Issue Type: Bug Components: storm-kafka Affects Versions: 0.11.0 Reporter: Michael Noll Assignee: Michael Noll Priority: Minor STORM-512 (KafkaBolt doesn't handle ticks properly) adds special-casing of tick tuples. What is missing in the patch is that the input tuple, when it is a tick tuple, should be properly acked like normal tuples. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-786: KafkaBolt should ack tick tuples
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/522 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: Question about Nimbus$Client
Hi, I had a quick look into it. JavaDoc can simply be added to the thrift code and is copied into generated Java and Python source code files. I modified storm.thrift and re-generated the code as described in DEVELOPER.md: cd storm-core/src sh genthrift.sh I pushed my changed to my git repo, in case you want to have a quick look: https://github.com/mjsax/storm I opened a JIRA, too: https://issues.apache.org/jira/browse/STORM-792 If you like it, I can open a pull request. Of course, it would be nice to add some more comments to all methods and interfaces. -Matthias On 04/20/2015 07:52 PM, Bobby Evans wrote: I totally agree there is a lot in the documentation that would be good to do. For the generated code I'm not totally sure how to make javadocs work. If you want to file a JIRA for this we can look at it. - Bobby On Saturday, April 18, 2015 9:49 AM, Matthias J. Sax mj...@informatik.hu-berlin.de wrote: Thanks! It is a petty, that this information in not documented via JavaDoc... That would make life much easier and would save time for everybody (= no need to ask and answer stupid questions on the mailing list) -Matthias On 04/17/2015 06:13 PM, Bobby Evans wrote: getTopology returns the compiled topology after nimbus has gotten its hands on it, so it has the ackers in it and the metrics consumers. getUserTopology returns the topology as the user submitted it. - Bobby On Friday, April 17, 2015 4:24 AM, Matthias J. Sax mj...@informatik.hu-berlin.de wrote: Dear all, the class backtype.storm.generated.Nimbus defines a nested class Nimbus$Cluster that offers the following two methods (both defined in Nimbus#Iface): public StormTopology getTopology(String id) throws NotAliveException, org.apache.thrift.TException; public StormTopology getUserTopology(String id) throws NotAliveException, org.apache.thrift.TException; What is the difference between both? I don't understand what the difference between a (regular?) topology and a user topology should be... From my understanding, there is only one type of topologies. Thanks for you help! -Matthias signature.asc Description: OpenPGP digital signature
[jira] [Commented] (STORM-766) Supervisor summary should include the version.
[ https://issues.apache.org/jira/browse/STORM-766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503625#comment-14503625 ] ASF GitHub Bot commented on STORM-766: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/529#issuecomment-94568937 Please upmerge, there is a merge conflict in supervisor.clj, but it does not look too complex. Supervisor summary should include the version. -- Key: STORM-766 URL: https://issues.apache.org/jira/browse/STORM-766 Project: Apache Storm Issue Type: Bug Affects Versions: 0.10.0 Reporter: Parth Brahmbhatt Assignee: Sanket Chintapalli Priority: Minor Fix For: 0.10.0 With the support for rolling upgrade, different nodes in the cluster can run different versions of storm. We should include the version in SupervisorSummary just like NimbusSummary so admins can identify nodes that needs upgrading/downgrading from UI. As part of this change I will also add a supervisor/log link in the ui. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-786) KafkaBolt should ack tick tuples
[ https://issues.apache.org/jira/browse/STORM-786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-786. --- Resolution: Fixed Fix Version/s: 0.11.0 KafkaBolt should ack tick tuples Key: STORM-786 URL: https://issues.apache.org/jira/browse/STORM-786 Project: Apache Storm Issue Type: Bug Components: storm-kafka Affects Versions: 0.11.0 Reporter: Michael Noll Assignee: Michael Noll Priority: Minor Fix For: 0.11.0 STORM-512 (KafkaBolt doesn't handle ticks properly) adds special-casing of tick tuples. What is missing in the patch is that the input tuple, when it is a tick tuple, should be properly acked like normal tuples. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: JIRA-792
GitHub user mjsax opened a pull request: https://github.com/apache/storm/pull/530 JIRA-792 You can merge this pull request into a Git repository by running: $ git pull https://github.com/mjsax/storm master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/530.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #530 commit ec1a8d3c804bf9edaf905e25ba9d88542a559cc7 Author: mjsax mj...@informatik.hu-berlin.de Date: 2015-04-20T20:16:30Z added JavaDoc for Nimubus --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Correcting a typo in Trident-API-Overview.md
GitHub user imadityab opened a pull request: https://github.com/apache/storm/pull/531 Correcting a typo in Trident-API-Overview.md Fixed a typo in the section explaining benefits of CombinerAggregator You can merge this pull request into a Git repository by running: $ git pull https://github.com/imadityab/storm master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/531.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #531 commit 6ac7d2dc72992ea14df8d789a1e6b41cc621a27a Author: imadityab imadit...@gmail.com Date: 2015-04-21T05:48:58Z Correcting a typo in Trident-API-Overview.md --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-770) NullPointerException in consumeBatchToCursor
[
https://issues.apache.org/jira/browse/STORM-770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503542#comment-14503542
]
Jungtaek Lim commented on STORM-770:
When you run patched version of Storm, please share log messages if any kind of
Can't transfer tuple - task value is null. tuple information ... messages
are available.
Please notice it will be WARN level.
NullPointerException in consumeBatchToCursor
Key: STORM-770
URL: https://issues.apache.org/jira/browse/STORM-770
Project: Apache Storm
Issue Type: Bug
Affects Versions: 0.9.2-incubating
Reporter: Stas Levin
We got the following exception after our topology had been up for ~2 days,
and I was wondering if it might be related.
Looks like task in mk-transfer-fn is null, making (.add remote
(TaskMessage. task (.serialize serializer tuple))) fail on NPE
(worker.clj:128, storm-core-0.9.2-incubating.jar)
java.lang.RuntimeException: java.lang.NullPointerException
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.disruptor$consume_loop_STAR_$fn__758.invoke(disruptor.clj:94)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_72]
Caused by: java.lang.NullPointerException: null
at clojure.lang.RT.intCast(RT.java:1087) ~[clojure-1.5.1.jar:na]
at
backtype.storm.daemon.worker$mk_transfer_fn$fn__5748.invoke(worker.clj:128)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.executor$start_batch_transfer_GT_worker_handler_BANG$fn__5483.invoke(executor.clj:256)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
... 6 common frames omitted,java.lang.RuntimeException:
java.lang.NullPointerException
Any ideas?
P.S.
Also saw it here:
http://mail-archives.apache.org/mod_mbox/storm-user/201501.mbox/%3CCABcMBhCusXXU=v1e66wfuatgyh1euqnd1siog65-tp8xlwx...@mail.gmail.com%3E
https://mail-archives.apache.org/mod_mbox/storm-user/201408.mbox/%3ccajuqm_4kxhsh2_x08ujuqr76m2c+dswp0fcijbmfcaeyqgs...@mail.gmail.com%3E
Comment from Bobby
http://mail-archives.apache.org/mod_mbox/storm-user/201501.mbox/%3c574363643.2791948.1420470097280.javamail.ya...@jws10027.mail.ne1.yahoo.com%3E
{quote}
What version of storm are you using? Are any of the bolts shell bolts?
There is a known
issue where this can happen if two shell bolts share an executor, because
they are multi-threaded.
- Bobby
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (STORM-792) Missing documentation in backtype.storm.generated.Nimbus
Matthias J. Sax created STORM-792: - Summary: Missing documentation in backtype.storm.generated.Nimbus Key: STORM-792 URL: https://issues.apache.org/jira/browse/STORM-792 Project: Apache Storm Issue Type: Documentation Reporter: Matthias J. Sax Priority: Minor Explain the difference between Nimbus$Client.getTopology(String id) and Nimbus$Client.getUserTopology(String id) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-643] KafkaUtils repeat fetch messages w...
Github user tpiscitell commented on the pull request:
https://github.com/apache/storm/pull/405#issuecomment-94448772
@miguno I don't think either of those JIRAs fixed this issue. The problem
here is that PartitionManager.fill() will always attempt to fetch any failed
tuples first:
```
// Are there failed tuples? If so, fetch those first.
if (had_failed) {
offset = failed.first();
} else {
offset = _emittedToOffset;
}
```
However, even with the patches for those two JIRAs, the `failed` list is
never pruned. Instead PartitionManager.fill() just update `_emitedToOffset` and
returns:
```
} catch (TopicOffsetOutOfRangeException e) {
_emittedToOffset = KafkaUtils.getOffset(_consumer,
_spoutConfig.topic, _partition.partition, _spoutConfig);
LOG.warn(Using new offset: {}, _emittedToOffset);
// fetch failed, so don't update the metrics
return;
}
```
Just to be called again, and the process repeats:
```
public EmitState next(SpoutOutputCollector collector) {
if (_waitingToEmit.isEmpty()) {
fill();
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-643) KafkaUtils repeatedly fetches messages whose offset is out of range
[
https://issues.apache.org/jira/browse/STORM-643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502768#comment-14502768
]
ASF GitHub Bot commented on STORM-643:
--
Github user tpiscitell commented on the pull request:
https://github.com/apache/storm/pull/405#issuecomment-94448772
@miguno I don't think either of those JIRAs fixed this issue. The problem
here is that PartitionManager.fill() will always attempt to fetch any failed
tuples first:
```
// Are there failed tuples? If so, fetch those first.
if (had_failed) {
offset = failed.first();
} else {
offset = _emittedToOffset;
}
```
However, even with the patches for those two JIRAs, the `failed` list is
never pruned. Instead PartitionManager.fill() just update `_emitedToOffset` and
returns:
```
} catch (TopicOffsetOutOfRangeException e) {
_emittedToOffset = KafkaUtils.getOffset(_consumer,
_spoutConfig.topic, _partition.partition, _spoutConfig);
LOG.warn(Using new offset: {}, _emittedToOffset);
// fetch failed, so don't update the metrics
return;
}
```
Just to be called again, and the process repeats:
```
public EmitState next(SpoutOutputCollector collector) {
if (_waitingToEmit.isEmpty()) {
fill();
}
```
KafkaUtils repeatedly fetches messages whose offset is out of range
---
Key: STORM-643
URL: https://issues.apache.org/jira/browse/STORM-643
Project: Apache Storm
Issue Type: Bug
Components: storm-kafka
Affects Versions: 0.9.2-incubating, 0.9.3
Reporter: Xin Wang
Assignee: Xin Wang
Priority: Minor
KafkaUtils repeat fetch messages which offset is out of range.
This happened when failed list(SortedSetLong failed) is not empty and some
offset in it is OutOfRange.
{code}
[worker-log]
2015-02-01 10:24:27.231+0800 s.k.KafkaUtils [WARN] Got fetch request with
offset out of range: [20919071816]; retrying with default start offset time
from configuration. configured start offset time: [-2]
2015-02-01 10:24:27.232+0800 s.k.PartitionManager [WARN] Using new offset:
20996130717
2015-02-01 10:24:27.333+0800 s.k.KafkaUtils [WARN] Got fetch request with
offset out of range: [20919071816]; retrying with default start offset time
from configuration. configured start offset time: [-2]
2015-02-01 10:24:27.334+0800 s.k.PartitionManager [WARN] Using new offset:
20996130717
...
{code}
[FIX]
{code}
storm.kafka.PartitionManager.fill():
...
try {
msgs = KafkaUtils.fetchMessages(_spoutConfig, _consumer, _partition,
offset);
} catch (UpdateOffsetException e) {
_emittedToOffset = KafkaUtils.getOffset(_consumer, _spoutConfig.topic,
_partition.partition, _spoutConfig);
LOG.warn(Using new offset: {}, _emittedToOffset);
// fetch failed, so don't update the metrics
//fix bug: remove this offset from failed list when it is OutOfRange
if (had_failed) {
failed.remove(offset);
}
return;
}
...
{code}
also: Log retrying with default start offset time from configuration.
configured start offset time: [-2] is incorrect.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-786) KafkaBolt should ack tick tuples
[ https://issues.apache.org/jira/browse/STORM-786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502457#comment-14502457 ] ASF GitHub Bot commented on STORM-786: -- Github user miguno commented on the pull request: https://github.com/apache/storm/pull/522#issuecomment-94378665 We already [received a +1](https://github.com/apache/storm/pull/275#issuecomment-94068063) from @nathanmarz. KafkaBolt should ack tick tuples Key: STORM-786 URL: https://issues.apache.org/jira/browse/STORM-786 Project: Apache Storm Issue Type: Bug Components: storm-kafka Affects Versions: 0.11.0 Reporter: Michael Noll Priority: Minor STORM-512 (KafkaBolt doesn't handle ticks properly) adds special-casing of tick tuples. What is missing in the patch is that the input tuple, when it is a tick tuple, should be properly acked like normal tuples. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-772) Tasts fail periodically with InterruptedException or InterruptedIOException
[ https://issues.apache.org/jira/browse/STORM-772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-772. --- Resolution: Fixed Fix Version/s: 0.11.0 Tasts fail periodically with InterruptedException or InterruptedIOException --- Key: STORM-772 URL: https://issues.apache.org/jira/browse/STORM-772 Project: Apache Storm Issue Type: Bug Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Labels: newbie Fix For: 0.11.0 Sometimes tests will fail when a local cluster is being shut down because a thread was interrupted. This seems to occur most around the disruptor queue consume-loop* in executor.clj. The error handler for that method does not deal with Interrupted type exceptions at all, instead it reports the error and shuts down the entire process. We should add in checks similar to other places that look for InterruptedException or InterruptedIOExcepion and ignore the Exception, but then shut down the thread. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-791: Storm UI displays maps in the confi...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/528#issuecomment-94467475 @HeartSaVioR yes, copy/paste comes out exactly as you see it. Even in the case of maps and lists with values in them there are new lines. I like the JSON because you can copy and paste it on the command line with a -c command. But the new lines make it a little difficult. If you think this is a blocker please let me know and I'll look for a work around, like having a copy button that does not include any extra white space. Of we could file a follow on JIRA for it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-791) Storm UI displays maps in the config incorrectly
[ https://issues.apache.org/jira/browse/STORM-791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502912#comment-14502912 ] ASF GitHub Bot commented on STORM-791: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/528#issuecomment-94467475 @HeartSaVioR yes, copy/paste comes out exactly as you see it. Even in the case of maps and lists with values in them there are new lines. I like the JSON because you can copy and paste it on the command line with a -c command. But the new lines make it a little difficult. If you think this is a blocker please let me know and I'll look for a work around, like having a copy button that does not include any extra white space. Of we could file a follow on JIRA for it. Storm UI displays maps in the config incorrectly Key: STORM-791 URL: https://issues.apache.org/jira/browse/STORM-791 Project: Apache Storm Issue Type: Bug Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans And it can be really confusing if a config is a list of strings or a single string with commas in it (as we ran into and had a painful time debugging). Lets format the values as JSON, so there is no ambiguity, and do some simple formatting of it so you can see everything around it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-789) Enhance topology context sent to multi-lang bolts and spouts
[
https://issues.apache.org/jira/browse/STORM-789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502950#comment-14502950
]
ASF GitHub Bot commented on STORM-789:
--
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/525#discussion_r28695178
--- Diff: storm-core/src/jvm/backtype/storm/task/TopologyContext.java ---
@@ -154,6 +154,18 @@ public Fields getThisOutputFields(String streamId) {
}
/**
+ * Gets the declared output fields for the specified stream id for the
component
+ * this task is a part of.
+ */
+public MapString, ListString getThisOutputFieldsForStreams() {
+ MapString, ListString streamToFields = new HashMapString,
ListString();
+ for (String stream : this.getThisStreams()) {
+ streamToFields.put(stream,
this.getThisOutputFields(stream).toList());
--- End diff --
and in a number of other places in the code too.
Enhance topology context sent to multi-lang bolts and spouts
Key: STORM-789
URL: https://issues.apache.org/jira/browse/STORM-789
Project: Apache Storm
Issue Type: Improvement
Reporter: Dan Blanchard
Assignee: Dan Blanchard
I'm about to submit a pull request that adds a lot more contextual
information to the context JSON dictionary that gets sent to multi-lang
bolts and spouts as part of their initial handshake. These additions will
make is so non-JVM developers have access to the sources, targets, and field
names for their bolts and spouts.
We will add support for this in streamparse (one of the more popular Python
libraries for Storm) as soon as this gets merged in.
Here are the details of what I've added:
- componentid: the component ID for this task, so you don't have to look it
up
in task-component by taskid
- streams: a list of all of the streams for this task
- stream-outputfields: a mapping from stream names to output fields for
that
stream
- stream-target-grouping: a mapping from stream names to the targets for
this
task to the kind of grouping they use.
- sources-grouping: a mapping from the component IDs of the sources to
their
grouping.
Note on groupings: groupings are either represented as a string (e.g.,
SHUFFLE) or a list of strings for field groupings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-789) Enhance topology context sent to multi-lang bolts and spouts
[
https://issues.apache.org/jira/browse/STORM-789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502949#comment-14502949
]
ASF GitHub Bot commented on STORM-789:
--
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/525#discussion_r28695109
--- Diff: storm-core/src/jvm/backtype/storm/task/TopologyContext.java ---
@@ -154,6 +154,18 @@ public Fields getThisOutputFields(String streamId) {
}
/**
+ * Gets the declared output fields for the specified stream id for the
component
+ * this task is a part of.
+ */
+public MapString, ListString getThisOutputFieldsForStreams() {
+ MapString, ListString streamToFields = new HashMapString,
ListString();
+ for (String stream : this.getThisStreams()) {
+ streamToFields.put(stream,
this.getThisOutputFields(stream).toList());
--- End diff --
minor nit: the white space appears to be off here.
Enhance topology context sent to multi-lang bolts and spouts
Key: STORM-789
URL: https://issues.apache.org/jira/browse/STORM-789
Project: Apache Storm
Issue Type: Improvement
Reporter: Dan Blanchard
Assignee: Dan Blanchard
I'm about to submit a pull request that adds a lot more contextual
information to the context JSON dictionary that gets sent to multi-lang
bolts and spouts as part of their initial handshake. These additions will
make is so non-JVM developers have access to the sources, targets, and field
names for their bolts and spouts.
We will add support for this in streamparse (one of the more popular Python
libraries for Storm) as soon as this gets merged in.
Here are the details of what I've added:
- componentid: the component ID for this task, so you don't have to look it
up
in task-component by taskid
- streams: a list of all of the streams for this task
- stream-outputfields: a mapping from stream names to output fields for
that
stream
- stream-target-grouping: a mapping from stream names to the targets for
this
task to the kind of grouping they use.
- sources-grouping: a mapping from the component IDs of the sources to
their
grouping.
Note on groupings: groupings are either represented as a string (e.g.,
SHUFFLE) or a list of strings for field groupings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-789) Enhance topology context sent to multi-lang bolts and spouts
[
https://issues.apache.org/jira/browse/STORM-789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502982#comment-14502982
]
ASF GitHub Bot commented on STORM-789:
--
Github user dan-blanchard commented on a diff in the pull request:
https://github.com/apache/storm/pull/525#discussion_r28696363
--- Diff: storm-core/src/jvm/backtype/storm/task/TopologyContext.java ---
@@ -154,6 +154,18 @@ public Fields getThisOutputFields(String streamId) {
}
/**
+ * Gets the declared output fields for the specified stream id for the
component
+ * this task is a part of.
+ */
+public MapString, ListString getThisOutputFieldsForStreams() {
+ MapString, ListString streamToFields = new HashMapString,
ListString();
+ for (String stream : this.getThisStreams()) {
+ streamToFields.put(stream,
this.getThisOutputFields(stream).toList());
--- End diff --
I'm not a big Java guy, so what's the appropriate spacing here? Tabs or
spaces? And how many?
Enhance topology context sent to multi-lang bolts and spouts
Key: STORM-789
URL: https://issues.apache.org/jira/browse/STORM-789
Project: Apache Storm
Issue Type: Improvement
Reporter: Dan Blanchard
Assignee: Dan Blanchard
I'm about to submit a pull request that adds a lot more contextual
information to the context JSON dictionary that gets sent to multi-lang
bolts and spouts as part of their initial handshake. These additions will
make is so non-JVM developers have access to the sources, targets, and field
names for their bolts and spouts.
We will add support for this in streamparse (one of the more popular Python
libraries for Storm) as soon as this gets merged in.
Here are the details of what I've added:
- componentid: the component ID for this task, so you don't have to look it
up
in task-component by taskid
- streams: a list of all of the streams for this task
- stream-outputfields: a mapping from stream names to output fields for
that
stream
- stream-target-grouping: a mapping from stream names to the targets for
this
task to the kind of grouping they use.
- sources-grouping: a mapping from the component IDs of the sources to
their
grouping.
Note on groupings: groupings are either represented as a string (e.g.,
SHUFFLE) or a list of strings for field groupings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: [STORM-789] Send more topology context to Mult...
Github user dan-blanchard commented on a diff in the pull request:
https://github.com/apache/storm/pull/525#discussion_r28696363
--- Diff: storm-core/src/jvm/backtype/storm/task/TopologyContext.java ---
@@ -154,6 +154,18 @@ public Fields getThisOutputFields(String streamId) {
}
/**
+ * Gets the declared output fields for the specified stream id for the
component
+ * this task is a part of.
+ */
+public MapString, ListString getThisOutputFieldsForStreams() {
+ MapString, ListString streamToFields = new HashMapString,
ListString();
+ for (String stream : this.getThisStreams()) {
+ streamToFields.put(stream,
this.getThisOutputFields(stream).toList());
--- End diff --
I'm not a big Java guy, so what's the appropriate spacing here? Tabs or
spaces? And how many?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-789] Send more topology context to Mult...
Github user dan-blanchard commented on a diff in the pull request:
https://github.com/apache/storm/pull/525#discussion_r28696404
--- Diff: storm-core/src/jvm/backtype/storm/task/TopologyContext.java ---
@@ -205,32 +217,65 @@ public Object getTaskData(String name) {
public void setExecutorData(String name, Object data) {
_executorData.put(name, data);
}
-
+
public Object getExecutorData(String name) {
return _executorData.get(name);
-}
-
+}
+
public void addTaskHook(ITaskHook hook) {
hook.prepare(_stormConf, this);
_hooks.add(hook);
}
-
+
public CollectionITaskHook getHooks() {
return _hooks;
}
+
+public Object groupingToJSONableObject(Grouping grouping) {
--- End diff --
Good point. I'll fix that. I also wasn't sure how much everyone would like
my made-up word JSONable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-766) Supervisor summary should include the version.
[
https://issues.apache.org/jira/browse/STORM-766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502869#comment-14502869
]
ASF GitHub Bot commented on STORM-766:
--
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/529#discussion_r28690338
--- Diff: storm-core/src/clj/backtype/storm/converter.clj ---
@@ -23,7 +25,9 @@
(if (.get_used_ports supervisor-info) (into [] (.get_used_ports
supervisor-info)))
(if (.get_meta supervisor-info) (into [] (.get_meta
supervisor-info)))
(if (.get_scheduler_meta supervisor-info) (into {}
(.get_scheduler_meta supervisor-info)))
- (.get_uptime_secs supervisor-info
+ (.get_uptime_secs supervisor-info)
+ (.get_version supervisor-info);;log
--- End diff --
Please clean up the comment, not sure what ;;log means, and move the
closing ')' to this line
Supervisor summary should include the version.
--
Key: STORM-766
URL: https://issues.apache.org/jira/browse/STORM-766
Project: Apache Storm
Issue Type: Bug
Affects Versions: 0.10.0
Reporter: Parth Brahmbhatt
Assignee: Sanket Chintapalli
Priority: Minor
Fix For: 0.10.0
With the support for rolling upgrade, different nodes in the cluster can run
different versions of storm. We should include the version in
SupervisorSummary just like NimbusSummary so admins can identify nodes that
needs upgrading/downgrading from UI.
As part of this change I will also add a supervisor/log link in the ui.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: JIRA STORM-766 (Include version info in the se...
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/529#discussion_r28690473
--- Diff: storm-core/src/storm.thrift ---
@@ -153,6 +153,7 @@ struct SupervisorSummary {
3: required i32 num_workers;
4: required i32 num_used_workers;
5: required string supervisor_id;
+ 6: required string version;
--- End diff --
This needs to be optional, to maintain rolling upgrade compatibility.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-766) Supervisor summary should include the version.
[
https://issues.apache.org/jira/browse/STORM-766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502872#comment-14502872
]
ASF GitHub Bot commented on STORM-766:
--
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/529#discussion_r28690473
--- Diff: storm-core/src/storm.thrift ---
@@ -153,6 +153,7 @@ struct SupervisorSummary {
3: required i32 num_workers;
4: required i32 num_used_workers;
5: required string supervisor_id;
+ 6: required string version;
--- End diff --
This needs to be optional, to maintain rolling upgrade compatibility.
Supervisor summary should include the version.
--
Key: STORM-766
URL: https://issues.apache.org/jira/browse/STORM-766
Project: Apache Storm
Issue Type: Bug
Affects Versions: 0.10.0
Reporter: Parth Brahmbhatt
Assignee: Sanket Chintapalli
Priority: Minor
Fix For: 0.10.0
With the support for rolling upgrade, different nodes in the cluster can run
different versions of storm. We should include the version in
SupervisorSummary just like NimbusSummary so admins can identify nodes that
needs upgrading/downgrading from UI.
As part of this change I will also add a supervisor/log link in the ui.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-791) Storm UI displays maps in the config incorrectly
[ https://issues.apache.org/jira/browse/STORM-791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503004#comment-14503004 ] ASF GitHub Bot commented on STORM-791: -- Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/528#issuecomment-94480822 @revans2 I don't think it's a blocker and it can be fixed when we think it'll be really necessary. Looks good to me again. Storm UI displays maps in the config incorrectly Key: STORM-791 URL: https://issues.apache.org/jira/browse/STORM-791 Project: Apache Storm Issue Type: Bug Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans And it can be really confusing if a config is a list of strings or a single string with commas in it (as we ran into and had a painful time debugging). Lets format the values as JSON, so there is no ambiguity, and do some simple formatting of it so you can see everything around it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-773: STORM-772: Transactional test timeo...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/526 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-773) backtype.storm.transactional-test fails periodically with timeout
[ https://issues.apache.org/jira/browse/STORM-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502866#comment-14502866 ] ASF GitHub Bot commented on STORM-773: -- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/526 backtype.storm.transactional-test fails periodically with timeout - Key: STORM-773 URL: https://issues.apache.org/jira/browse/STORM-773 Project: Apache Storm Issue Type: Bug Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Fix For: 0.11.0 Attachments: failure.txt, success.txt I'm not totally sure what is happening here, but fairly frequently now on my mac running JDK8 backtype.storm.transactional-test will timeout. test-transactional-topology-restart seems to be the test in there that is getting the timeouts. I made some modifications to the test to just run that one test case, and to turn topology.debug on. I captured examples of it working and failing. I'll attach the logs files shortly. I am have not really had much time to dig into this, so I am not totally sure what is happening here. I can see from the logs that on the first run of the topology the failure case only emits 10 batches, where as the successful case outputs many more. On the second run of the topology the failure case starts off at batch 11, but does not go beyond it. Where as the successful case keeps going. I'll try to find some time to look into it more, but I'm not sure how much time I will have in the near future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-789] Send more topology context to Mult...
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/525#discussion_r28695109
--- Diff: storm-core/src/jvm/backtype/storm/task/TopologyContext.java ---
@@ -154,6 +154,18 @@ public Fields getThisOutputFields(String streamId) {
}
/**
+ * Gets the declared output fields for the specified stream id for the
component
+ * this task is a part of.
+ */
+public MapString, ListString getThisOutputFieldsForStreams() {
+ MapString, ListString streamToFields = new HashMapString,
ListString();
+ for (String stream : this.getThisStreams()) {
+ streamToFields.put(stream,
this.getThisOutputFields(stream).toList());
--- End diff --
minor nit: the white space appears to be off here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Resolved] (STORM-773) backtype.storm.transactional-test fails periodically with timeout
[ https://issues.apache.org/jira/browse/STORM-773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-773. --- Resolution: Fixed Fix Version/s: 0.11.0 backtype.storm.transactional-test fails periodically with timeout - Key: STORM-773 URL: https://issues.apache.org/jira/browse/STORM-773 Project: Apache Storm Issue Type: Bug Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Fix For: 0.11.0 Attachments: failure.txt, success.txt I'm not totally sure what is happening here, but fairly frequently now on my mac running JDK8 backtype.storm.transactional-test will timeout. test-transactional-topology-restart seems to be the test in there that is getting the timeouts. I made some modifications to the test to just run that one test case, and to turn topology.debug on. I captured examples of it working and failing. I'll attach the logs files shortly. I am have not really had much time to dig into this, so I am not totally sure what is happening here. I can see from the logs that on the first run of the topology the failure case only emits 10 batches, where as the successful case outputs many more. On the second run of the topology the failure case starts off at batch 11, but does not go beyond it. Where as the successful case keeps going. I'll try to find some time to look into it more, but I'm not sure how much time I will have in the near future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-791: Storm UI displays maps in the confi...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/528#issuecomment-94464428 @HeartSaVioR I didn't try it, but I will. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-790 Log task is null instead of let wo...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/527#issuecomment-94471958 @HeartSaVioR Looking at the code prior to https://github.com/apache/storm/commit/861a92eab8740cfc0f83ac4d7ade9a2ab04a8b9b null routing was silently ignored. I'm not sure that was on purpose though. I am OK merging this in, but I really would like to understand why the destination task is showing up as null occasionally, but I see that you addressed that in the comments here https://issues.apache.org/jira/browse/STORM-770?focusedCommentId=14499225page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14499225 So I am +1 for the change. At least now we explicitly handle the situation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-766) Supervisor summary should include the version.
[ https://issues.apache.org/jira/browse/STORM-766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502879#comment-14502879 ] ASF GitHub Bot commented on STORM-766: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/529#issuecomment-94460394 Overall the code looks good. The trick with the rolling upgrade is that we want to be sure that an old nimbus can work with a new supervisor and a new nimbus can work with an old supervisor too. Similarly for the client. An old client should be able to work with a new nimbus and a new client should, in most cases, work with an old nimbus. Perhaps the simples way to do this is to provide a default value for version which is set to something like VERSION NOT PROVIDED. Supervisor summary should include the version. -- Key: STORM-766 URL: https://issues.apache.org/jira/browse/STORM-766 Project: Apache Storm Issue Type: Bug Affects Versions: 0.10.0 Reporter: Parth Brahmbhatt Assignee: Sanket Chintapalli Priority: Minor Fix For: 0.10.0 With the support for rolling upgrade, different nodes in the cluster can run different versions of storm. We should include the version in SupervisorSummary just like NimbusSummary so admins can identify nodes that needs upgrading/downgrading from UI. As part of this change I will also add a supervisor/log link in the ui. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-791) Storm UI displays maps in the config incorrectly
[ https://issues.apache.org/jira/browse/STORM-791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502899#comment-14502899 ] ASF GitHub Bot commented on STORM-791: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/528#issuecomment-94464428 @HeartSaVioR I didn't try it, but I will. Storm UI displays maps in the config incorrectly Key: STORM-791 URL: https://issues.apache.org/jira/browse/STORM-791 Project: Apache Storm Issue Type: Bug Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans And it can be really confusing if a config is a list of strings or a single string with commas in it (as we ran into and had a painful time debugging). Lets format the values as JSON, so there is no ambiguity, and do some simple formatting of it so you can see everything around it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-790) Log task id is null instead of let worker died (NPE in consumeBatchToCursor)
[ https://issues.apache.org/jira/browse/STORM-790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502942#comment-14502942 ] ASF GitHub Bot commented on STORM-790: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/527#issuecomment-94471958 @HeartSaVioR Looking at the code prior to https://github.com/apache/storm/commit/861a92eab8740cfc0f83ac4d7ade9a2ab04a8b9b null routing was silently ignored. I'm not sure that was on purpose though. I am OK merging this in, but I really would like to understand why the destination task is showing up as null occasionally, but I see that you addressed that in the comments here https://issues.apache.org/jira/browse/STORM-770?focusedCommentId=14499225page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14499225 So I am +1 for the change. At least now we explicitly handle the situation. Log task id is null instead of let worker died (NPE in consumeBatchToCursor) -- Key: STORM-790 URL: https://issues.apache.org/jira/browse/STORM-790 Project: Apache Storm Issue Type: Bug Affects Versions: 0.9.2-incubating, 0.9.3, 0.10.0, 0.9.4, 0.11.0 Reporter: Jungtaek Lim Assignee: Jungtaek Lim In STORM-770, some users have observed that worker suddenly died with NPE in consumeBatchToCursor(). Looks like it can occur when task in mk-transfer-fn is null. It may not be an issue before 0.9.2-incubating, since worker just ignores that tuple. Please see https://issues.apache.org/jira/browse/STORM-770?focusedCommentId=14496199page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14496199. Before finding root cause of this issue, it would be better to let worker not killed by this issue but just log with WARN or ERROR level. It really makes sense cause before 0.9.2 Storm silently ignores tuple, and with Guaranteeing Message Processing, after timed-out tuple will be replayed. (It isn't applied to non-ack) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-789) Enhance topology context sent to multi-lang bolts and spouts
[
https://issues.apache.org/jira/browse/STORM-789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502958#comment-14502958
]
ASF GitHub Bot commented on STORM-789:
--
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/525#discussion_r28695524
--- Diff: storm-core/src/jvm/backtype/storm/task/TopologyContext.java ---
@@ -205,32 +217,65 @@ public Object getTaskData(String name) {
public void setExecutorData(String name, Object data) {
_executorData.put(name, data);
}
-
+
public Object getExecutorData(String name) {
return _executorData.get(name);
-}
-
+}
+
public void addTaskHook(ITaskHook hook) {
hook.prepare(_stormConf, this);
_hooks.add(hook);
}
-
+
public CollectionITaskHook getHooks() {
return _hooks;
}
+
+public Object groupingToJSONableObject(Grouping grouping) {
--- End diff --
This feels like it should be a static private method. I don't see any
reason for anyone else to call this.
Enhance topology context sent to multi-lang bolts and spouts
Key: STORM-789
URL: https://issues.apache.org/jira/browse/STORM-789
Project: Apache Storm
Issue Type: Improvement
Reporter: Dan Blanchard
Assignee: Dan Blanchard
I'm about to submit a pull request that adds a lot more contextual
information to the context JSON dictionary that gets sent to multi-lang
bolts and spouts as part of their initial handshake. These additions will
make is so non-JVM developers have access to the sources, targets, and field
names for their bolts and spouts.
We will add support for this in streamparse (one of the more popular Python
libraries for Storm) as soon as this gets merged in.
Here are the details of what I've added:
- componentid: the component ID for this task, so you don't have to look it
up
in task-component by taskid
- streams: a list of all of the streams for this task
- stream-outputfields: a mapping from stream names to output fields for
that
stream
- stream-target-grouping: a mapping from stream names to the targets for
this
task to the kind of grouping they use.
- sources-grouping: a mapping from the component IDs of the sources to
their
grouping.
Note on groupings: groupings are either represented as a string (e.g.,
SHUFFLE) or a list of strings for field groupings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: JIRA STORM-766 (Include version info in the se...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/529#issuecomment-94460394 Overall the code looks good. The trick with the rolling upgrade is that we want to be sure that an old nimbus can work with a new supervisor and a new nimbus can work with an old supervisor too. Similarly for the client. An old client should be able to work with a new nimbus and a new client should, in most cases, work with an old nimbus. Perhaps the simples way to do this is to provide a default value for version which is set to something like VERSION NOT PROVIDED. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-789] Send more topology context to Mult...
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/525#discussion_r28695178
--- Diff: storm-core/src/jvm/backtype/storm/task/TopologyContext.java ---
@@ -154,6 +154,18 @@ public Fields getThisOutputFields(String streamId) {
}
/**
+ * Gets the declared output fields for the specified stream id for the
component
+ * this task is a part of.
+ */
+public MapString, ListString getThisOutputFieldsForStreams() {
+ MapString, ListString streamToFields = new HashMapString,
ListString();
+ for (String stream : this.getThisStreams()) {
+ streamToFields.put(stream,
this.getThisOutputFields(stream).toList());
--- End diff --
and in a number of other places in the code too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-789] Send more topology context to Mult...
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/525#discussion_r28695524
--- Diff: storm-core/src/jvm/backtype/storm/task/TopologyContext.java ---
@@ -205,32 +217,65 @@ public Object getTaskData(String name) {
public void setExecutorData(String name, Object data) {
_executorData.put(name, data);
}
-
+
public Object getExecutorData(String name) {
return _executorData.get(name);
-}
-
+}
+
public void addTaskHook(ITaskHook hook) {
hook.prepare(_stormConf, this);
_hooks.add(hook);
}
-
+
public CollectionITaskHook getHooks() {
return _hooks;
}
+
+public Object groupingToJSONableObject(Grouping grouping) {
--- End diff --
This feels like it should be a static private method. I don't see any
reason for anyone else to call this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-791: Storm UI displays maps in the confi...
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/528#issuecomment-94480822 @revans2 I don't think it's a blocker and it can be fixed when we think it'll be really necessary. Looks good to me again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: Concurrent requests to DRPC
Ok so DRPC does work this way, but not in a scalable way. The DRPC spout and return bolt are non-blocking in the DRPC server. This means that the topology can scale very well with tasks that take a long time to process, but you may need to adjust timeouts in your topology and on the DRPC servers to make this work. The issue is with the client. Each client makes a blocking call to the DRPC server, so if you want to have multiple requests outstanding you need multiple client instances. The DRPC server also has a limited number of threads so after those threads run out new client connections will be rejected. I did a prototype a while ago using the Servlet 3.0 async API to allow the server to scale much better, but because storm still uses the very old jetty 6 http server which doesn't support Servlet 3.0 I have not done much with it in quite a while. - Bobby On Saturday, April 18, 2015 3:35 PM, Douglas Alan d...@alum.mit.edu wrote: I'm not a developer of Storm, but I asked this question on us...@storm.apache.org, and no one seemed to know the answer. (Or if they did, they did not see fit to respond.) So I'm going to try here instead. Hopefully you will forgive the intrusion. I want to make a Storm topology than handles DRPC requests, and I want the requests to be handled concurrently. I.e., if a DRPC request comes into the topology at time T0 that takes a long time to complete (let's say 100 seconds), I don't want this request to block another request that comes in at T0 + 1 sec and will only take 1 second to complete. I.e, I'd like to get the result for the second request at time T0 + 2 sec or T0 + 3 sec or so, and not at time T0 + 101 sec. In my experiments so far, I have not been able to get this to work. In fact, if I send a request to a Storm DRPC topology while it is still working on a previous request, the second request is not even buffered. I just end up receiving an exception when I send the second request. Is there a way to do what I want? Or is Storm DRPC inherently sequential? Thanks for your help, |oug
[GitHub] storm pull request: JIRA STORM-766 (Include version info in the se...
Github user redsanket commented on the pull request: https://github.com/apache/storm/pull/529#issuecomment-94522345 I have modified the code as per the specifications. Please let me know if there are any specifications that have to be met. Thanks Sanket --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: Upgrading Storm to use Netty 4.0 or higher
We have not really explored going to netty 4.0. I know at some point we will probably have to switch over to use it, but for the time being most of the dependencies that we have seen still use netty 3.X, but if you have code to use netty 4.x and want to contribute I would be happy to review it and see what we can do to support that. Even if it involves doing some shading to support it. - Bobby On Friday, April 17, 2015 10:48 PM, Julian Stephen julian.step...@gmail.com wrote: Hi All, I am quite curious to know if there is any interest or activity around upgrading Storm to use Netty 4.0 or higher. From what I understand, Storm (at least till the 0.10.0 dev branch I checked out) still depends on Netty 3.x. and Netty 4.x is not compatible with 3.x code Link http://netty.io/wiki/new-and-noteworthy-in-4.0.html. I could not find any related issues in the Apache Storm Jira, and is curious to know if anyone has explored implications and impact of such a change. Regards, Julian
[GitHub] storm pull request: [STORM-789] Send more topology context to Mult...
Github user dan-blanchard commented on a diff in the pull request:
https://github.com/apache/storm/pull/525#discussion_r28710870
--- Diff: storm-core/src/jvm/backtype/storm/task/TopologyContext.java ---
@@ -205,32 +217,65 @@ public Object getTaskData(String name) {
public void setExecutorData(String name, Object data) {
_executorData.put(name, data);
}
-
+
public Object getExecutorData(String name) {
return _executorData.get(name);
-}
-
+}
+
public void addTaskHook(ITaskHook hook) {
hook.prepare(_stormConf, this);
_hooks.add(hook);
}
-
+
public CollectionITaskHook getHooks() {
return _hooks;
}
+
+public Object groupingToJSONableObject(Grouping grouping) {
+ if (grouping.is_set_fields()) {
+ return grouping.get_fields();
+ } else {
+ return grouping.getSetField().toString();
+ }
--- End diff --
Okay, that's reasonable to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-766) Supervisor summary should include the version.
[ https://issues.apache.org/jira/browse/STORM-766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503290#comment-14503290 ] ASF GitHub Bot commented on STORM-766: -- Github user redsanket commented on the pull request: https://github.com/apache/storm/pull/529#issuecomment-94522345 I have modified the code as per the specifications. Please let me know if there are any specifications that have to be met. Thanks Sanket Supervisor summary should include the version. -- Key: STORM-766 URL: https://issues.apache.org/jira/browse/STORM-766 Project: Apache Storm Issue Type: Bug Affects Versions: 0.10.0 Reporter: Parth Brahmbhatt Assignee: Sanket Chintapalli Priority: Minor Fix For: 0.10.0 With the support for rolling upgrade, different nodes in the cluster can run different versions of storm. We should include the version in SupervisorSummary just like NimbusSummary so admins can identify nodes that needs upgrading/downgrading from UI. As part of this change I will also add a supervisor/log link in the ui. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-789] Send more topology context to Mult...
Github user dan-blanchard commented on a diff in the pull request:
https://github.com/apache/storm/pull/525#discussion_r28699109
--- Diff: storm-core/src/jvm/backtype/storm/task/TopologyContext.java ---
@@ -205,32 +217,65 @@ public Object getTaskData(String name) {
public void setExecutorData(String name, Object data) {
_executorData.put(name, data);
}
-
+
public Object getExecutorData(String name) {
return _executorData.get(name);
-}
-
+}
+
public void addTaskHook(ITaskHook hook) {
hook.prepare(_stormConf, this);
_hooks.add(hook);
}
-
+
public CollectionITaskHook getHooks() {
return _hooks;
}
+
+public Object groupingToJSONableObject(Grouping grouping) {
+ if (grouping.is_set_fields()) {
+ return grouping.get_fields();
+ } else {
+ return grouping.getSetField().toString();
+ }
--- End diff --
I understand wanting to make something that's easily extensible in the
future, but the example you showed wouldn't ever happen, because only the
fields grouping specifies fields. If you think it should return a dictionary
that always contains `type` and `fields`, we'd get the following for shuffle
groupings:
```json
{
type: SHUFFLE,
fields: null
}
```
and
```json
{
type: FIELDS,
fields: [a, b]
}
```
for fields groupings.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-788] Fix key for process latencies
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/524#issuecomment-94498604 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-788) UI Component Page Input shows execute latency where it should show process latency
[ https://issues.apache.org/jira/browse/STORM-788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503120#comment-14503120 ] ASF GitHub Bot commented on STORM-788: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/524#issuecomment-94498604 +1 UI Component Page Input shows execute latency where it should show process latency -- Key: STORM-788 URL: https://issues.apache.org/jira/browse/STORM-788 Project: Apache Storm Issue Type: Bug Affects Versions: 0.9.2-incubating Reporter: Derek Dagit Assignee: Derek Dagit Priority: Minor The execute latency value is show in place of the process latency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-791) Storm UI displays maps in the config incorrectly
[ https://issues.apache.org/jira/browse/STORM-791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503233#comment-14503233 ] ASF GitHub Bot commented on STORM-791: -- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/528 Storm UI displays maps in the config incorrectly Key: STORM-791 URL: https://issues.apache.org/jira/browse/STORM-791 Project: Apache Storm Issue Type: Bug Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans And it can be really confusing if a config is a list of strings or a single string with commas in it (as we ran into and had a painful time debugging). Lets format the values as JSON, so there is no ambiguity, and do some simple formatting of it so you can see everything around it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-789) Enhance topology context sent to multi-lang bolts and spouts
[
https://issues.apache.org/jira/browse/STORM-789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503281#comment-14503281
]
ASF GitHub Bot commented on STORM-789:
--
Github user dan-blanchard commented on a diff in the pull request:
https://github.com/apache/storm/pull/525#discussion_r28710870
--- Diff: storm-core/src/jvm/backtype/storm/task/TopologyContext.java ---
@@ -205,32 +217,65 @@ public Object getTaskData(String name) {
public void setExecutorData(String name, Object data) {
_executorData.put(name, data);
}
-
+
public Object getExecutorData(String name) {
return _executorData.get(name);
-}
-
+}
+
public void addTaskHook(ITaskHook hook) {
hook.prepare(_stormConf, this);
_hooks.add(hook);
}
-
+
public CollectionITaskHook getHooks() {
return _hooks;
}
+
+public Object groupingToJSONableObject(Grouping grouping) {
+ if (grouping.is_set_fields()) {
+ return grouping.get_fields();
+ } else {
+ return grouping.getSetField().toString();
+ }
--- End diff --
Okay, that's reasonable to me.
Enhance topology context sent to multi-lang bolts and spouts
Key: STORM-789
URL: https://issues.apache.org/jira/browse/STORM-789
Project: Apache Storm
Issue Type: Improvement
Reporter: Dan Blanchard
Assignee: Dan Blanchard
I'm about to submit a pull request that adds a lot more contextual
information to the context JSON dictionary that gets sent to multi-lang
bolts and spouts as part of their initial handshake. These additions will
make is so non-JVM developers have access to the sources, targets, and field
names for their bolts and spouts.
We will add support for this in streamparse (one of the more popular Python
libraries for Storm) as soon as this gets merged in.
Here are the details of what I've added:
- componentid: the component ID for this task, so you don't have to look it
up
in task-component by taskid
- streams: a list of all of the streams for this task
- stream-outputfields: a mapping from stream names to output fields for
that
stream
- stream-target-grouping: a mapping from stream names to the targets for
this
task to the kind of grouping they use.
- sources-grouping: a mapping from the component IDs of the sources to
their
grouping.
Note on groupings: groupings are either represented as a string (e.g.,
SHUFFLE) or a list of strings for field groupings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Question about Nimbus$Client
I totally agree there is a lot in the documentation that would be good to do. For the generated code I'm not totally sure how to make javadocs work. If you want to file a JIRA for this we can look at it. - Bobby On Saturday, April 18, 2015 9:49 AM, Matthias J. Sax mj...@informatik.hu-berlin.de wrote: Thanks! It is a petty, that this information in not documented via JavaDoc... That would make life much easier and would save time for everybody (= no need to ask and answer stupid questions on the mailing list) -Matthias On 04/17/2015 06:13 PM, Bobby Evans wrote: getTopology returns the compiled topology after nimbus has gotten its hands on it, so it has the ackers in it and the metrics consumers. getUserTopology returns the topology as the user submitted it. - Bobby On Friday, April 17, 2015 4:24 AM, Matthias J. Sax mj...@informatik.hu-berlin.de wrote: Dear all, the class backtype.storm.generated.Nimbus defines a nested class Nimbus$Cluster that offers the following two methods (both defined in Nimbus#Iface): public StormTopology getTopology(String id) throws NotAliveException, org.apache.thrift.TException; public StormTopology getUserTopology(String id) throws NotAliveException, org.apache.thrift.TException; What is the difference between both? I don't understand what the difference between a (regular?) topology and a user topology should be... From my understanding, there is only one type of topologies. Thanks for you help! -Matthias
[jira] [Commented] (STORM-789) Enhance topology context sent to multi-lang bolts and spouts
[
https://issues.apache.org/jira/browse/STORM-789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503225#comment-14503225
]
ASF GitHub Bot commented on STORM-789:
--
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/525#discussion_r28708308
--- Diff: storm-core/src/jvm/backtype/storm/task/TopologyContext.java ---
@@ -205,32 +217,65 @@ public Object getTaskData(String name) {
public void setExecutorData(String name, Object data) {
_executorData.put(name, data);
}
-
+
public Object getExecutorData(String name) {
return _executorData.get(name);
-}
-
+}
+
public void addTaskHook(ITaskHook hook) {
hook.prepare(_stormConf, this);
_hooks.add(hook);
}
-
+
public CollectionITaskHook getHooks() {
return _hooks;
}
+
+public Object groupingToJSONableObject(Grouping grouping) {
+ if (grouping.is_set_fields()) {
+ return grouping.get_fields();
+ } else {
+ return grouping.getSetField().toString();
+ }
--- End diff --
I was thinking it would look kind of like.
```{type:SHUFFLE}```
instead of having the fields be null. type would be the only required
field, but I am open to other ideas.
Enhance topology context sent to multi-lang bolts and spouts
Key: STORM-789
URL: https://issues.apache.org/jira/browse/STORM-789
Project: Apache Storm
Issue Type: Improvement
Reporter: Dan Blanchard
Assignee: Dan Blanchard
I'm about to submit a pull request that adds a lot more contextual
information to the context JSON dictionary that gets sent to multi-lang
bolts and spouts as part of their initial handshake. These additions will
make is so non-JVM developers have access to the sources, targets, and field
names for their bolts and spouts.
We will add support for this in streamparse (one of the more popular Python
libraries for Storm) as soon as this gets merged in.
Here are the details of what I've added:
- componentid: the component ID for this task, so you don't have to look it
up
in task-component by taskid
- streams: a list of all of the streams for this task
- stream-outputfields: a mapping from stream names to output fields for
that
stream
- stream-target-grouping: a mapping from stream names to the targets for
this
task to the kind of grouping they use.
- sources-grouping: a mapping from the component IDs of the sources to
their
grouping.
Note on groupings: groupings are either represented as a string (e.g.,
SHUFFLE) or a list of strings for field groupings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-791: Storm UI displays maps in the confi...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/528 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-789] Send more topology context to Mult...
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/525#discussion_r28708308
--- Diff: storm-core/src/jvm/backtype/storm/task/TopologyContext.java ---
@@ -205,32 +217,65 @@ public Object getTaskData(String name) {
public void setExecutorData(String name, Object data) {
_executorData.put(name, data);
}
-
+
public Object getExecutorData(String name) {
return _executorData.get(name);
-}
-
+}
+
public void addTaskHook(ITaskHook hook) {
hook.prepare(_stormConf, this);
_hooks.add(hook);
}
-
+
public CollectionITaskHook getHooks() {
return _hooks;
}
+
+public Object groupingToJSONableObject(Grouping grouping) {
+ if (grouping.is_set_fields()) {
+ return grouping.get_fields();
+ } else {
+ return grouping.getSetField().toString();
+ }
--- End diff --
I was thinking it would look kind of like.
```{type:SHUFFLE}```
instead of having the fields be null. type would be the only required
field, but I am open to other ideas.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Resolved] (STORM-791) Storm UI displays maps in the config incorrectly
[ https://issues.apache.org/jira/browse/STORM-791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-791. --- Resolution: Fixed Fix Version/s: 0.11.0 I merged this into master. If copy/paste becomes an issue we can file a JIRA at that time. Storm UI displays maps in the config incorrectly Key: STORM-791 URL: https://issues.apache.org/jira/browse/STORM-791 Project: Apache Storm Issue Type: Bug Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Fix For: 0.11.0 And it can be really confusing if a config is a list of strings or a single string with commas in it (as we ran into and had a painful time debugging). Lets format the values as JSON, so there is no ambiguity, and do some simple formatting of it so you can see everything around it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-789) Enhance topology context sent to multi-lang bolts and spouts
[ https://issues.apache.org/jira/browse/STORM-789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503100#comment-14503100 ] ASF GitHub Bot commented on STORM-789: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/525#issuecomment-94496477 @dan-blanchard thanks for all of the work you have done. I did a pass through the code and I had a few nits about whitespace (not important) and I was curious what you though about changing the way the grouping conversion works. If you don't really want to change it, I am fine with that. My only other comment would be around documentation. Please update https://github.com/apache/storm/blob/master/docs/documentation/Multilang-protocol.md to include the new information, and mark that these changes are a part of 0.11.0. Enhance topology context sent to multi-lang bolts and spouts Key: STORM-789 URL: https://issues.apache.org/jira/browse/STORM-789 Project: Apache Storm Issue Type: Improvement Reporter: Dan Blanchard Assignee: Dan Blanchard I'm about to submit a pull request that adds a lot more contextual information to the context JSON dictionary that gets sent to multi-lang bolts and spouts as part of their initial handshake. These additions will make is so non-JVM developers have access to the sources, targets, and field names for their bolts and spouts. We will add support for this in streamparse (one of the more popular Python libraries for Storm) as soon as this gets merged in. Here are the details of what I've added: - componentid: the component ID for this task, so you don't have to look it up in task-component by taskid - streams: a list of all of the streams for this task - stream-outputfields: a mapping from stream names to output fields for that stream - stream-target-grouping: a mapping from stream names to the targets for this task to the kind of grouping they use. - sources-grouping: a mapping from the component IDs of the sources to their grouping. Note on groupings: groupings are either represented as a string (e.g., SHUFFLE) or a list of strings for field groupings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-789) Enhance topology context sent to multi-lang bolts and spouts
[
https://issues.apache.org/jira/browse/STORM-789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503036#comment-14503036
]
ASF GitHub Bot commented on STORM-789:
--
Github user dan-blanchard commented on a diff in the pull request:
https://github.com/apache/storm/pull/525#discussion_r28699109
--- Diff: storm-core/src/jvm/backtype/storm/task/TopologyContext.java ---
@@ -205,32 +217,65 @@ public Object getTaskData(String name) {
public void setExecutorData(String name, Object data) {
_executorData.put(name, data);
}
-
+
public Object getExecutorData(String name) {
return _executorData.get(name);
-}
-
+}
+
public void addTaskHook(ITaskHook hook) {
hook.prepare(_stormConf, this);
_hooks.add(hook);
}
-
+
public CollectionITaskHook getHooks() {
return _hooks;
}
+
+public Object groupingToJSONableObject(Grouping grouping) {
+ if (grouping.is_set_fields()) {
+ return grouping.get_fields();
+ } else {
+ return grouping.getSetField().toString();
+ }
--- End diff --
I understand wanting to make something that's easily extensible in the
future, but the example you showed wouldn't ever happen, because only the
fields grouping specifies fields. If you think it should return a dictionary
that always contains `type` and `fields`, we'd get the following for shuffle
groupings:
```json
{
type: SHUFFLE,
fields: null
}
```
and
```json
{
type: FIELDS,
fields: [a, b]
}
```
for fields groupings.
Enhance topology context sent to multi-lang bolts and spouts
Key: STORM-789
URL: https://issues.apache.org/jira/browse/STORM-789
Project: Apache Storm
Issue Type: Improvement
Reporter: Dan Blanchard
Assignee: Dan Blanchard
I'm about to submit a pull request that adds a lot more contextual
information to the context JSON dictionary that gets sent to multi-lang
bolts and spouts as part of their initial handshake. These additions will
make is so non-JVM developers have access to the sources, targets, and field
names for their bolts and spouts.
We will add support for this in streamparse (one of the more popular Python
libraries for Storm) as soon as this gets merged in.
Here are the details of what I've added:
- componentid: the component ID for this task, so you don't have to look it
up
in task-component by taskid
- streams: a list of all of the streams for this task
- stream-outputfields: a mapping from stream names to output fields for
that
stream
- stream-target-grouping: a mapping from stream names to the targets for
this
task to the kind of grouping they use.
- sources-grouping: a mapping from the component IDs of the sources to
their
grouping.
Note on groupings: groupings are either represented as a string (e.g.,
SHUFFLE) or a list of strings for field groupings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: [STORM-789] Send more topology context to Mult...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/525#issuecomment-94496477 @dan-blanchard thanks for all of the work you have done. I did a pass through the code and I had a few nits about whitespace (not important) and I was curious what you though about changing the way the grouping conversion works. If you don't really want to change it, I am fine with that. My only other comment would be around documentation. Please update https://github.com/apache/storm/blob/master/docs/documentation/Multilang-protocol.md to include the new information, and mark that these changes are a part of 0.11.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-788] Fix key for process latencies
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/524 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-788) UI Component Page Input shows execute latency where it should show process latency
[ https://issues.apache.org/jira/browse/STORM-788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503122#comment-14503122 ] ASF GitHub Bot commented on STORM-788: -- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/524 UI Component Page Input shows execute latency where it should show process latency -- Key: STORM-788 URL: https://issues.apache.org/jira/browse/STORM-788 Project: Apache Storm Issue Type: Bug Affects Versions: 0.9.2-incubating Reporter: Derek Dagit Assignee: Derek Dagit Priority: Minor The execute latency value is show in place of the process latency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-788) UI Component Page Input shows execute latency where it should show process latency
[ https://issues.apache.org/jira/browse/STORM-788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-788. --- Resolution: Fixed Fix Version/s: 0.11.0 Thanks [~dagit], I merged this into master, if you think this should go into 0.10.X please let me know and I'll try to cherry-pick it in. UI Component Page Input shows execute latency where it should show process latency -- Key: STORM-788 URL: https://issues.apache.org/jira/browse/STORM-788 Project: Apache Storm Issue Type: Bug Affects Versions: 0.9.2-incubating Reporter: Derek Dagit Assignee: Derek Dagit Priority: Minor Fix For: 0.11.0 The execute latency value is show in place of the process latency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-583) Add spout and bolt implementation for Azure Eventhubs
[ https://issues.apache.org/jira/browse/STORM-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503462#comment-14503462 ] P. Taylor Goetz commented on STORM-583: --- IP Clearance form submitted: http://incubator.apache.org/ip-clearance/storm-azure-eventhubs.html Add spout and bolt implementation for Azure Eventhubs - Key: STORM-583 URL: https://issues.apache.org/jira/browse/STORM-583 Project: Apache Storm Issue Type: Bug Affects Versions: 0.9.3 Reporter: shanyu zhao Assignee: shanyu zhao Fix For: 0.10.0 Attachments: Storm-EventHubsDesign.docx, storm-eventhubs.tar.gz Add spout and bolt implementations for Azure Eventhubs - a messaging service that supports AMQP protocol. Just like storm-kafka/storm-hbase, we need to add the project to the /external folder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-789] Send more topology context to Mult...
Github user dan-blanchard commented on the pull request:
https://github.com/apache/storm/pull/525#issuecomment-94553135
@revans2 Thanks for the review. I haven't updated the documentation yet,
but I did fix the grouping-related stuff, now we get output like:
```python
{u'task-component':
{u'20': u'__acker',
u'21': u'__acker',
u'22': u'__acker',
u'1': u'1',
u'3': u'__acker',
u'2': u'2',
u'5': u'__acker',
u'4': u'__acker',
u'7': u'__acker',
u'6': u'__acker',
u'9': u'__acker',
u'8': u'__acker',
u'11': u'__acker',
u'10': u'__acker',
u'13': u'__acker',
u'12': u'__acker',
u'15': u'__acker',
u'14': u'__acker',
u'17': u'__acker',
u'16': u'__acker',
u'19': u'__acker',
u'18': u'__acker'},
u'stream-target-grouping': {u'default': {u'2': {u'fields': [u'word'],
u'type': u'FIELDS'}}},
u'streams': [u'default'],
u'stream-outputfields': {u'default': [u'word']},
u'taskid': 1,
u'source-stream-grouping': {},
u'componentid': u'1'}
```
for something with a fields grouping and
```python
{u'task-component':
{u'20': u'__acker',
u'21': u'__acker',
u'22': u'__acker',
u'1': u'1',
u'3': u'__acker',
u'2': u'2',
u'5': u'__acker',
u'4': u'__acker',
u'7': u'__acker',
u'6': u'__acker',
u'9': u'__acker',
u'8': u'__acker',
u'11': u'__acker',
u'10': u'__acker',
u'13': u'__acker',
u'12': u'__acker',
u'15': u'__acker',
u'14': u'__acker',
u'17': u'__acker',
u'16': u'__acker',
u'19': u'__acker',
u'18': u'__acker'},
u'stream-target-grouping': {u'default': {u'2': {u'type': u'SHUFFLE'}}},
u'streams': [u'default'],
u'stream-outputfields': {u'default': [u'word']},
u'taskid': 1,
u'source-stream-grouping': {},
u'componentid': u'1'}
```
for shuffle.
I'll get the documentation updated soon.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-789) Enhance topology context sent to multi-lang bolts and spouts
[
https://issues.apache.org/jira/browse/STORM-789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503504#comment-14503504
]
ASF GitHub Bot commented on STORM-789:
--
Github user dan-blanchard commented on the pull request:
https://github.com/apache/storm/pull/525#issuecomment-94553135
@revans2 Thanks for the review. I haven't updated the documentation yet,
but I did fix the grouping-related stuff, now we get output like:
```python
{u'task-component':
{u'20': u'__acker',
u'21': u'__acker',
u'22': u'__acker',
u'1': u'1',
u'3': u'__acker',
u'2': u'2',
u'5': u'__acker',
u'4': u'__acker',
u'7': u'__acker',
u'6': u'__acker',
u'9': u'__acker',
u'8': u'__acker',
u'11': u'__acker',
u'10': u'__acker',
u'13': u'__acker',
u'12': u'__acker',
u'15': u'__acker',
u'14': u'__acker',
u'17': u'__acker',
u'16': u'__acker',
u'19': u'__acker',
u'18': u'__acker'},
u'stream-target-grouping': {u'default': {u'2': {u'fields': [u'word'],
u'type': u'FIELDS'}}},
u'streams': [u'default'],
u'stream-outputfields': {u'default': [u'word']},
u'taskid': 1,
u'source-stream-grouping': {},
u'componentid': u'1'}
```
for something with a fields grouping and
```python
{u'task-component':
{u'20': u'__acker',
u'21': u'__acker',
u'22': u'__acker',
u'1': u'1',
u'3': u'__acker',
u'2': u'2',
u'5': u'__acker',
u'4': u'__acker',
u'7': u'__acker',
u'6': u'__acker',
u'9': u'__acker',
u'8': u'__acker',
u'11': u'__acker',
u'10': u'__acker',
u'13': u'__acker',
u'12': u'__acker',
u'15': u'__acker',
u'14': u'__acker',
u'17': u'__acker',
u'16': u'__acker',
u'19': u'__acker',
u'18': u'__acker'},
u'stream-target-grouping': {u'default': {u'2': {u'type': u'SHUFFLE'}}},
u'streams': [u'default'],
u'stream-outputfields': {u'default': [u'word']},
u'taskid': 1,
u'source-stream-grouping': {},
u'componentid': u'1'}
```
for shuffle.
I'll get the documentation updated soon.
Enhance topology context sent to multi-lang bolts and spouts
Key: STORM-789
URL: https://issues.apache.org/jira/browse/STORM-789
Project: Apache Storm
Issue Type: Improvement
Reporter: Dan Blanchard
Assignee: Dan Blanchard
I'm about to submit a pull request that adds a lot more contextual
information to the context JSON dictionary that gets sent to multi-lang
bolts and spouts as part of their initial handshake. These additions will
make is so non-JVM developers have access to the sources, targets, and field
names for their bolts and spouts.
We will add support for this in streamparse (one of the more popular Python
libraries for Storm) as soon as this gets merged in.
Here are the details of what I've added:
- componentid: the component ID for this task, so you don't have to look it
up
in task-component by taskid
- streams: a list of all of the streams for this task
- stream-outputfields: a mapping from stream names to output fields for
that
stream
- stream-target-grouping: a mapping from stream names to the targets for
this
task to the kind of grouping they use.
- sources-grouping: a mapping from the component IDs of the sources to
their
grouping.
Note on groupings: groupings are either represented as a string (e.g.,
SHUFFLE) or a list of strings for field groupings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
