[GitHub] storm pull request: Fix minor grammar bug in javadoc
GitHub user markdav opened a pull request: https://github.com/apache/storm/pull/535 Fix minor grammar bug in javadoc Internet grammar police and apostrophes! You can merge this pull request into a Git repository by running: $ git pull https://github.com/markdav/storm patch-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/535.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #535 commit c84f20514e81cfdd3c48b5cf89dfe6a9c4d328bb Author: Mark Davis Date: 2015-04-24T06:48:50Z Fix minor grammar bug in javadoc Internet grammar police and apostrophes! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-436) Clean up Clojure namespace declarations
[ https://issues.apache.org/jira/browse/STORM-436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14510541#comment-14510541 ] Daniel Compton commented on STORM-436: -- Can I get some feedback on the pull request to make sure I'm on the right track and people are happy with the style? https://github.com/apache/storm/pull/534/ > Clean up Clojure namespace declarations > --- > > Key: STORM-436 > URL: https://issues.apache.org/jira/browse/STORM-436 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 0.9.2-incubating >Reporter: Daniel Compton >Priority: Minor > Labels: newbie > > Some of the Clojure namespace declarations in the storm project are messy and > non-idiomatic. > https://github.com/apache/incubator-storm/blob/master/storm-core/src/clj/backtype/storm/ui/core.clj#L18-L38 > is a good example of this. > There are a few things I'd like to improve: > 1. Coalesce multiple use/require/import's into a single use/require/import. > 2. Order imports use, require, import. > 3. Optionally, replacing use with some mix of > * [... :refer :all] > * Referring just the vars that are used > * Qualifying the namespace imports > Is there a reason why the namespaces were done in the way they have been? > What would be the preferred way to do 3? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-436) Clean up Clojure namespace declarations
[ https://issues.apache.org/jira/browse/STORM-436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14510535#comment-14510535 ] ASF GitHub Bot commented on STORM-436: -- GitHub user danielcompton opened a pull request: https://github.com/apache/storm/pull/534 [STORM-436] Clean up ns declarations This is a WIP, I'm wanting to get feedback on the style before going too far. You can merge this pull request into a Git repository by running: $ git pull https://github.com/danielcompton/storm clean-up-ns Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/534.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #534 commit 912eea4ff02818ef319b3897ba892ea5cd529c2a Author: Daniel Compton Date: 2015-04-24T06:27:08Z [STORM-436] Clean up ns declarations > Clean up Clojure namespace declarations > --- > > Key: STORM-436 > URL: https://issues.apache.org/jira/browse/STORM-436 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 0.9.2-incubating >Reporter: Daniel Compton >Priority: Minor > Labels: newbie > > Some of the Clojure namespace declarations in the storm project are messy and > non-idiomatic. > https://github.com/apache/incubator-storm/blob/master/storm-core/src/clj/backtype/storm/ui/core.clj#L18-L38 > is a good example of this. > There are a few things I'd like to improve: > 1. Coalesce multiple use/require/import's into a single use/require/import. > 2. Order imports use, require, import. > 3. Optionally, replacing use with some mix of > * [... :refer :all] > * Referring just the vars that are used > * Qualifying the namespace imports > Is there a reason why the namespaces were done in the way they have been? > What would be the preferred way to do 3? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-436] Clean up ns declarations
GitHub user danielcompton opened a pull request: https://github.com/apache/storm/pull/534 [STORM-436] Clean up ns declarations This is a WIP, I'm wanting to get feedback on the style before going too far. You can merge this pull request into a Git repository by running: $ git pull https://github.com/danielcompton/storm clean-up-ns Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/534.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #534 commit 912eea4ff02818ef319b3897ba892ea5cd529c2a Author: Daniel Compton Date: 2015-04-24T06:27:08Z [STORM-436] Clean up ns declarations --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (STORM-798) storm-kafka tests are failing intermittently
Jungtaek Lim created STORM-798:
--
Summary: storm-kafka tests are failing intermittently
Key: STORM-798
URL: https://issues.apache.org/jira/browse/STORM-798
Project: Apache Storm
Issue Type: Bug
Components: storm-kafka
Reporter: Jungtaek Lim
Priority: Minor
I'm trying to adopt Travis CI and observed failures about storm-kafka test.
https://travis-ci.org/apache/storm/jobs/59795630
Full log is here.
https://s3.amazonaws.com/archive.travis-ci.org/jobs/59795629/log.txt
{noformat}
Tests in error:
testKeyValue(storm.kafka.TridentKafkaTest): Could not send message with key =
key-123 to topic = test
generateTuplesWithKeyAndKeyValueScheme(storm.kafka.KafkaUtilsTest): Error
fetching data from [Partition{host=localhost:52047, partition=0}] for topic
[testTopic]: [OFFSET_OUT_OF_RANGE]
{noformat}
Please note that tests run with VM, which could be slow at moment or while
testing.
It would be better to let tests pass from slow box without random failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-704) Apply Travis CI
[ https://issues.apache.org/jira/browse/STORM-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509952#comment-14509952 ] ASF GitHub Bot commented on STORM-704: -- Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/486#issuecomment-95734570 Infra setup seems to be completed. I can see current PR's build status. > Apply Travis CI > --- > > Key: STORM-704 > URL: https://issues.apache.org/jira/browse/STORM-704 > Project: Apache Storm > Issue Type: Improvement > Environment: Travis CI >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Minor > > Now Apache Storm takes advantage of Github, we can apply Travis CI to some > more advantages. > - Build matrix > -- Travis CI supports various JDK versions (openjdk6, openjdk7, oraclejdk7, > oraclejdk8), and it can be tested separately. > - Build automatically > -- pushed new commits, any new PRs > - Integrated with Github > -- Contributors can see his/her PR breaks compilation / test in some minutes. > -- If he/she adds commits to PR, Travis builds it automatically and update > build result. > Please see [https://travis-ci.org/xetorthio/jedis] for example. > There're some hurdles applying Travis CI to Apache Storm project, but we can > overcome these and finally get great CI. > Current hurdles > - asfgit should manage Travis CI setup for the first time > -- other Apache projects already did it by requesting it to INFRA > --- ex. [https://issues.apache.org/jira/browse/INFRA-6161] > - Travis CI restricts stdout with 4M which is too small for Storm maven > output. > -- Change log level in tests to WARN > - In storm-core, we can't see tests failure information on stdout cause it > just prints 'clojure failed'. > -- We need to find a way to upload surefire / clojure tests report files to > somewhere, and uploaded files should be visible easily. > --- Travis supports uploading artifacts to S3 by > [https://github.com/travis-ci/artifacts], but S3 is not free. > -- I'm not familiar with Clojure, but can we print tests summary with clojure > tests as same as Java junit tests? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-704 Apply Travis CI to Apache Storm Proj...
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/486#issuecomment-95734570 Infra setup seems to be completed. I can see current PR's build status. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-704) Apply Travis CI
[
https://issues.apache.org/jira/browse/STORM-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509949#comment-14509949
]
ASF GitHub Bot commented on STORM-704:
--
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/486#issuecomment-95734278
https://travis-ci.org/HeartSaVioR/storm/builds/59794019 succeed and
https://travis-ci.org/HeartSaVioR/storm/builds/59795636 failed, which means
failed tests could be a kind of random failures or relied on machine power.
First one is DisruptorQueueTest, as you already know.
Failed tests:
testMessageDisorder(backtype.storm.utils.DisruptorQueueTest): We expect to
receive first published message first, but received null expected:<1> but
was:
Second one is Kafka Tests, which I met intermittently from my dev. machine.
testKeyValue(storm.kafka.TridentKafkaTest): Could not send message with
key = key-123 to topic = test
generateTuplesWithKeyAndKeyValueScheme(storm.kafka.KafkaUtilsTest): Error
fetching data from [Partition{host=localhost:52047, partition=0}] for topic
[testTopic]: [OFFSET_OUT_OF_RANGE]
> Apply Travis CI
> ---
>
> Key: STORM-704
> URL: https://issues.apache.org/jira/browse/STORM-704
> Project: Apache Storm
> Issue Type: Improvement
> Environment: Travis CI
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>Priority: Minor
>
> Now Apache Storm takes advantage of Github, we can apply Travis CI to some
> more advantages.
> - Build matrix
> -- Travis CI supports various JDK versions (openjdk6, openjdk7, oraclejdk7,
> oraclejdk8), and it can be tested separately.
> - Build automatically
> -- pushed new commits, any new PRs
> - Integrated with Github
> -- Contributors can see his/her PR breaks compilation / test in some minutes.
> -- If he/she adds commits to PR, Travis builds it automatically and update
> build result.
> Please see [https://travis-ci.org/xetorthio/jedis] for example.
> There're some hurdles applying Travis CI to Apache Storm project, but we can
> overcome these and finally get great CI.
> Current hurdles
> - asfgit should manage Travis CI setup for the first time
> -- other Apache projects already did it by requesting it to INFRA
> --- ex. [https://issues.apache.org/jira/browse/INFRA-6161]
> - Travis CI restricts stdout with 4M which is too small for Storm maven
> output.
> -- Change log level in tests to WARN
> - In storm-core, we can't see tests failure information on stdout cause it
> just prints 'clojure failed'.
> -- We need to find a way to upload surefire / clojure tests report files to
> somewhere, and uploaded files should be visible easily.
> --- Travis supports uploading artifacts to S3 by
> [https://github.com/travis-ci/artifacts], but S3 is not free.
> -- I'm not familiar with Clojure, but can we print tests summary with clojure
> tests as same as Java junit tests?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-704 Apply Travis CI to Apache Storm Proj...
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/486#issuecomment-95734278
https://travis-ci.org/HeartSaVioR/storm/builds/59794019 succeed and
https://travis-ci.org/HeartSaVioR/storm/builds/59795636 failed, which means
failed tests could be a kind of random failures or relied on machine power.
First one is DisruptorQueueTest, as you already know.
Failed tests:
testMessageDisorder(backtype.storm.utils.DisruptorQueueTest): We expect to
receive first published message first, but received null expected:<1> but
was:
Second one is Kafka Tests, which I met intermittently from my dev. machine.
testKeyValue(storm.kafka.TridentKafkaTest): Could not send message with
key = key-123 to topic = test
generateTuplesWithKeyAndKeyValueScheme(storm.kafka.KafkaUtilsTest): Error
fetching data from [Partition{host=localhost:52047, partition=0}] for topic
[testTopic]: [OFFSET_OUT_OF_RANGE]
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-704) Apply Travis CI
[ https://issues.apache.org/jira/browse/STORM-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509931#comment-14509931 ] ASF GitHub Bot commented on STORM-704: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/486#issuecomment-95732261 Yes, so the issue that we have run into before, and may be the same now, is that ruby does not come with JSON support by default on ubuntu. But that was a much older version or ruby and ubuntu. So who knows. > Apply Travis CI > --- > > Key: STORM-704 > URL: https://issues.apache.org/jira/browse/STORM-704 > Project: Apache Storm > Issue Type: Improvement > Environment: Travis CI >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Minor > > Now Apache Storm takes advantage of Github, we can apply Travis CI to some > more advantages. > - Build matrix > -- Travis CI supports various JDK versions (openjdk6, openjdk7, oraclejdk7, > oraclejdk8), and it can be tested separately. > - Build automatically > -- pushed new commits, any new PRs > - Integrated with Github > -- Contributors can see his/her PR breaks compilation / test in some minutes. > -- If he/she adds commits to PR, Travis builds it automatically and update > build result. > Please see [https://travis-ci.org/xetorthio/jedis] for example. > There're some hurdles applying Travis CI to Apache Storm project, but we can > overcome these and finally get great CI. > Current hurdles > - asfgit should manage Travis CI setup for the first time > -- other Apache projects already did it by requesting it to INFRA > --- ex. [https://issues.apache.org/jira/browse/INFRA-6161] > - Travis CI restricts stdout with 4M which is too small for Storm maven > output. > -- Change log level in tests to WARN > - In storm-core, we can't see tests failure information on stdout cause it > just prints 'clojure failed'. > -- We need to find a way to upload surefire / clojure tests report files to > somewhere, and uploaded files should be visible easily. > --- Travis supports uploading artifacts to S3 by > [https://github.com/travis-ci/artifacts], but S3 is not free. > -- I'm not familiar with Clojure, but can we print tests summary with clojure > tests as same as Java junit tests? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-704 Apply Travis CI to Apache Storm Proj...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/486#issuecomment-95732261 Yes, so the issue that we have run into before, and may be the same now, is that ruby does not come with JSON support by default on ubuntu. But that was a much older version or ruby and ubuntu. So who knows. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-770) NullPointerException in consumeBatchToCursor
[
https://issues.apache.org/jira/browse/STORM-770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509858#comment-14509858
]
Jungtaek Lim commented on STORM-770:
I am confused, too. I'm beginner to clojure, I don't even think about it could
be possible.
> NullPointerException in consumeBatchToCursor
>
>
> Key: STORM-770
> URL: https://issues.apache.org/jira/browse/STORM-770
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 0.9.2-incubating
>Reporter: Stas Levin
>
> We got the following exception after our topology had been up for ~2 days,
> and I was wondering if it might be related.
> Looks like "task" in "mk-transfer-fn" is null, making "(.add remote
> (TaskMessage. task (.serialize serializer tuple)))" fail on NPE
> (worker.clj:128, storm-core-0.9.2-incubating.jar)
> java.lang.RuntimeException: java.lang.NullPointerException
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$consume_loop_STAR_$fn__758.invoke(disruptor.clj:94)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_72]
> Caused by: java.lang.NullPointerException: null
> at clojure.lang.RT.intCast(RT.java:1087) ~[clojure-1.5.1.jar:na]
> at
> backtype.storm.daemon.worker$mk_transfer_fn$fn__5748.invoke(worker.clj:128)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.executor$start_batch_transfer_GT_worker_handler_BANG$fn__5483.invoke(executor.clj:256)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> ... 6 common frames omitted,java.lang.RuntimeException:
> java.lang.NullPointerException
> Any ideas?
> P.S.
> Also saw it here:
> http://mail-archives.apache.org/mod_mbox/storm-user/201501.mbox/%3CCABcMBhCusXXU=v1e66wfuatgyh1euqnd1siog65-tp8xlwx...@mail.gmail.com%3E
> https://mail-archives.apache.org/mod_mbox/storm-user/201408.mbox/%3ccajuqm_4kxhsh2_x08ujuqr76m2c+dswp0fcijbmfcaeyqgs...@mail.gmail.com%3E
> Comment from Bobby
> http://mail-archives.apache.org/mod_mbox/storm-user/201501.mbox/%3c574363643.2791948.1420470097280.javamail.ya...@jws10027.mail.ne1.yahoo.com%3E
> {quote}
> What version of storm are you using? Are any of the bolts shell bolts?
> There is a known
> issue where this can happen if two shell bolts share an executor, because
> they are multi-threaded.
> - Bobby
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (STORM-797) Disruptor Queue message order issue
Kishor Patil created STORM-797:
--
Summary: Disruptor Queue message order issue
Key: STORM-797
URL: https://issues.apache.org/jira/browse/STORM-797
Project: Apache Storm
Issue Type: Bug
Reporter: Kishor Patil
We notice following ??DisruptorQueueTest.testMessageDisorder?? unit test fails
intermittently:
{code}
java.lang.AssertionError: We expect to receive first published message first,
but received null expected:<1> but was:
DisruptorQueueTest.testMessageDisorder:60 We expect to receive first
published message first, but received null expected:<1> but was:
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-796) The "error" command isn't supported on ShellSpout
[ https://issues.apache.org/jira/browse/STORM-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509854#comment-14509854 ] Jungtaek Lim commented on STORM-796: Fix Version/s is for marking version which version resolves it. Current master is for 0.11.0, and if you wish to apply it to 0.10.0, it would be better to explain it should be fixed faster. > The "error" command isn't supported on ShellSpout > - > > Key: STORM-796 > URL: https://issues.apache.org/jira/browse/STORM-796 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 0.9.4 >Reporter: Andrew Montalenti >Assignee: Andrew Montalenti > > The `ShellBolt` can handle the "error" command, as shown in this file in > Storm source code: > https://github.com/apache/storm/blob/2dd7a9426e5634211f14cf5c4e10e021d3420c3c/storm-core/src/jvm/backtype/storm/task/ShellBolt.java#L330 > But, `ShellSpout` does not actually have a handler for "error". > https://github.com/apache/storm/blob/2dd7a9426e5634211f14cf5c4e10e021d3420c3c/storm-core/src/jvm/backtype/storm/spout/ShellSpout.java#L153-L175 > The symptoms a multi-lang user will see here is that if their Spout throws an > error and their multi-lang implementation sends an "error" command up to the > ShellSpout, the ShellSpout will respond saying that it doesn't recognize the > "error" command, and thus it will crash (while swallowing the exception > thrown by the underlying multi-lang component). > I am about to open a PR on Github that fixes this. > Originally reported on the streamparse project in this Github issue: > https://github.com/Parsely/streamparse/issues/121 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-796) The "error" command isn't supported on ShellSpout
[ https://issues.apache.org/jira/browse/STORM-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509850#comment-14509850 ] ASF GitHub Bot commented on STORM-796: -- Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/533#issuecomment-95723096 LGTM. > The "error" command isn't supported on ShellSpout > - > > Key: STORM-796 > URL: https://issues.apache.org/jira/browse/STORM-796 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 0.9.4 >Reporter: Andrew Montalenti >Assignee: Andrew Montalenti > > The `ShellBolt` can handle the "error" command, as shown in this file in > Storm source code: > https://github.com/apache/storm/blob/2dd7a9426e5634211f14cf5c4e10e021d3420c3c/storm-core/src/jvm/backtype/storm/task/ShellBolt.java#L330 > But, `ShellSpout` does not actually have a handler for "error". > https://github.com/apache/storm/blob/2dd7a9426e5634211f14cf5c4e10e021d3420c3c/storm-core/src/jvm/backtype/storm/spout/ShellSpout.java#L153-L175 > The symptoms a multi-lang user will see here is that if their Spout throws an > error and their multi-lang implementation sends an "error" command up to the > ShellSpout, the ShellSpout will respond saying that it doesn't recognize the > "error" command, and thus it will crash (while swallowing the exception > thrown by the underlying multi-lang component). > I am about to open a PR on Github that fixes this. > Originally reported on the streamparse project in this Github issue: > https://github.com/Parsely/streamparse/issues/121 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-796] Add support for "error" command in...
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/533#issuecomment-95723096 LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: Why is Storm Config not modifiable?
Thanks for your explanations. IHMO, from a user point of view, the interface of open()/prepare() is not well designed. It specifies to hand-in a Java-Map, and you would not expect that a Map is not modifiable. Would it be possible to change the parameter type to clojure.lang.APersistentMap ? This would expose the persistent nature of the map to the user. Not sure if this is possible, due to compatibility issues... It's just a thought. -Matthias On 04/23/2015 06:10 PM, Bobby Evans wrote: > Yes you could all assoc on the Map, if you cast the map to an IPersistentMap > and call the assoc method on it, but if you do it blindly without checking > they actual type of the Map passed in you may run into issues in the future, > if we do change some internals away from java. > - Bobby > > > > On Thursday, April 23, 2015 9:07 AM, Jeremy Heiler > wrote: > > > On Thu, Apr 23, 2015 at 4:42 AM, Matthias J. Sax < > mj...@informatik.hu-berlin.de> wrote: > >> In my case, I use a class hierarchy with abstract spouts/bolts and >> multiple concrete spouts/bolts. The abstract class uses some default >> values (if not provided in the given config) similar to the derived >> classes. However, some derived classes overwrite the default values for >> the abstract class, ie, if no value is given in the config, some derived >> classes set the values and other do not. Additionally, open()/prepare() >> of the abstract class is called, too, and set the default value if the >> value was neither provided by the user, not by the derived class. >> >> > The idiomatic way to update an immutable field like this in Clojure would > be to use an atom. Here you could store the conf in an AtomicReference > field, call .assoc on the map to update it, and .compareAndSet it on the > AtomicReference. I'm not sure you'll need this, but I figured I'd mention > it as how you would use a PersistentHashMap in your situation. > > > > signature.asc Description: OpenPGP digital signature
[jira] [Updated] (STORM-796) The "error" command isn't supported on ShellSpout
[ https://issues.apache.org/jira/browse/STORM-796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-796: --- Fix Version/s: (was: 0.10.0) > The "error" command isn't supported on ShellSpout > - > > Key: STORM-796 > URL: https://issues.apache.org/jira/browse/STORM-796 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 0.9.4 >Reporter: Andrew Montalenti >Assignee: Andrew Montalenti > > The `ShellBolt` can handle the "error" command, as shown in this file in > Storm source code: > https://github.com/apache/storm/blob/2dd7a9426e5634211f14cf5c4e10e021d3420c3c/storm-core/src/jvm/backtype/storm/task/ShellBolt.java#L330 > But, `ShellSpout` does not actually have a handler for "error". > https://github.com/apache/storm/blob/2dd7a9426e5634211f14cf5c4e10e021d3420c3c/storm-core/src/jvm/backtype/storm/spout/ShellSpout.java#L153-L175 > The symptoms a multi-lang user will see here is that if their Spout throws an > error and their multi-lang implementation sends an "error" command up to the > ShellSpout, the ShellSpout will respond saying that it doesn't recognize the > "error" command, and thus it will crash (while swallowing the exception > thrown by the underlying multi-lang component). > I am about to open a PR on Github that fixes this. > Originally reported on the streamparse project in this Github issue: > https://github.com/Parsely/streamparse/issues/121 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-704) Apply Travis CI
[ https://issues.apache.org/jira/browse/STORM-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509833#comment-14509833 ] ASF GitHub Bot commented on STORM-704: -- Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/486#issuecomment-95721463 Last build is successful except multilang tests. https://travis-ci.org/HeartSaVioR/storm/builds/59788403 > Apply Travis CI > --- > > Key: STORM-704 > URL: https://issues.apache.org/jira/browse/STORM-704 > Project: Apache Storm > Issue Type: Improvement > Environment: Travis CI >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Minor > > Now Apache Storm takes advantage of Github, we can apply Travis CI to some > more advantages. > - Build matrix > -- Travis CI supports various JDK versions (openjdk6, openjdk7, oraclejdk7, > oraclejdk8), and it can be tested separately. > - Build automatically > -- pushed new commits, any new PRs > - Integrated with Github > -- Contributors can see his/her PR breaks compilation / test in some minutes. > -- If he/she adds commits to PR, Travis builds it automatically and update > build result. > Please see [https://travis-ci.org/xetorthio/jedis] for example. > There're some hurdles applying Travis CI to Apache Storm project, but we can > overcome these and finally get great CI. > Current hurdles > - asfgit should manage Travis CI setup for the first time > -- other Apache projects already did it by requesting it to INFRA > --- ex. [https://issues.apache.org/jira/browse/INFRA-6161] > - Travis CI restricts stdout with 4M which is too small for Storm maven > output. > -- Change log level in tests to WARN > - In storm-core, we can't see tests failure information on stdout cause it > just prints 'clojure failed'. > -- We need to find a way to upload surefire / clojure tests report files to > somewhere, and uploaded files should be visible easily. > --- Travis supports uploading artifacts to S3 by > [https://github.com/travis-ci/artifacts], but S3 is not free. > -- I'm not familiar with Clojure, but can we print tests summary with clojure > tests as same as Java junit tests? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-704 Apply Travis CI to Apache Storm Proj...
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/486#issuecomment-95721463 Last build is successful except multilang tests. https://travis-ci.org/HeartSaVioR/storm/builds/59788403 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-704) Apply Travis CI
[ https://issues.apache.org/jira/browse/STORM-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509826#comment-14509826 ] ASF GitHub Bot commented on STORM-704: -- Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/486#issuecomment-95720834 Mailed to Travis CI support to how can I get python, ruby, nodejs installed. If we really need to install these manually, I'll lend a debug VM from Travis CI. > Apply Travis CI > --- > > Key: STORM-704 > URL: https://issues.apache.org/jira/browse/STORM-704 > Project: Apache Storm > Issue Type: Improvement > Environment: Travis CI >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Minor > > Now Apache Storm takes advantage of Github, we can apply Travis CI to some > more advantages. > - Build matrix > -- Travis CI supports various JDK versions (openjdk6, openjdk7, oraclejdk7, > oraclejdk8), and it can be tested separately. > - Build automatically > -- pushed new commits, any new PRs > - Integrated with Github > -- Contributors can see his/her PR breaks compilation / test in some minutes. > -- If he/she adds commits to PR, Travis builds it automatically and update > build result. > Please see [https://travis-ci.org/xetorthio/jedis] for example. > There're some hurdles applying Travis CI to Apache Storm project, but we can > overcome these and finally get great CI. > Current hurdles > - asfgit should manage Travis CI setup for the first time > -- other Apache projects already did it by requesting it to INFRA > --- ex. [https://issues.apache.org/jira/browse/INFRA-6161] > - Travis CI restricts stdout with 4M which is too small for Storm maven > output. > -- Change log level in tests to WARN > - In storm-core, we can't see tests failure information on stdout cause it > just prints 'clojure failed'. > -- We need to find a way to upload surefire / clojure tests report files to > somewhere, and uploaded files should be visible easily. > --- Travis supports uploading artifacts to S3 by > [https://github.com/travis-ci/artifacts], but S3 is not free. > -- I'm not familiar with Clojure, but can we print tests summary with clojure > tests as same as Java junit tests? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-704 Apply Travis CI to Apache Storm Proj...
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/486#issuecomment-95720834 Mailed to Travis CI support to how can I get python, ruby, nodejs installed. If we really need to install these manually, I'll lend a debug VM from Travis CI. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Issue Comment Deleted] (STORM-796) The "error" command isn't supported on ShellSpout
[ https://issues.apache.org/jira/browse/STORM-796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Montalenti updated STORM-796: Comment: was deleted (was: The pull request was submitted here: https://github.com/apache/storm/pull/533) > The "error" command isn't supported on ShellSpout > - > > Key: STORM-796 > URL: https://issues.apache.org/jira/browse/STORM-796 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 0.9.4 >Reporter: Andrew Montalenti >Assignee: Andrew Montalenti > Fix For: 0.10.0 > > > The `ShellBolt` can handle the "error" command, as shown in this file in > Storm source code: > https://github.com/apache/storm/blob/2dd7a9426e5634211f14cf5c4e10e021d3420c3c/storm-core/src/jvm/backtype/storm/task/ShellBolt.java#L330 > But, `ShellSpout` does not actually have a handler for "error". > https://github.com/apache/storm/blob/2dd7a9426e5634211f14cf5c4e10e021d3420c3c/storm-core/src/jvm/backtype/storm/spout/ShellSpout.java#L153-L175 > The symptoms a multi-lang user will see here is that if their Spout throws an > error and their multi-lang implementation sends an "error" command up to the > ShellSpout, the ShellSpout will respond saying that it doesn't recognize the > "error" command, and thus it will crash (while swallowing the exception > thrown by the underlying multi-lang component). > I am about to open a PR on Github that fixes this. > Originally reported on the streamparse project in this Github issue: > https://github.com/Parsely/streamparse/issues/121 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-796) The "error" command isn't supported on ShellSpout
[ https://issues.apache.org/jira/browse/STORM-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509791#comment-14509791 ] Andrew Montalenti commented on STORM-796: - The pull request was submitted here: https://github.com/apache/storm/pull/533 > The "error" command isn't supported on ShellSpout > - > > Key: STORM-796 > URL: https://issues.apache.org/jira/browse/STORM-796 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 0.9.4 >Reporter: Andrew Montalenti >Assignee: Andrew Montalenti > Fix For: 0.10.0 > > > The `ShellBolt` can handle the "error" command, as shown in this file in > Storm source code: > https://github.com/apache/storm/blob/2dd7a9426e5634211f14cf5c4e10e021d3420c3c/storm-core/src/jvm/backtype/storm/task/ShellBolt.java#L330 > But, `ShellSpout` does not actually have a handler for "error". > https://github.com/apache/storm/blob/2dd7a9426e5634211f14cf5c4e10e021d3420c3c/storm-core/src/jvm/backtype/storm/spout/ShellSpout.java#L153-L175 > The symptoms a multi-lang user will see here is that if their Spout throws an > error and their multi-lang implementation sends an "error" command up to the > ShellSpout, the ShellSpout will respond saying that it doesn't recognize the > "error" command, and thus it will crash (while swallowing the exception > thrown by the underlying multi-lang component). > I am about to open a PR on Github that fixes this. > Originally reported on the streamparse project in this Github issue: > https://github.com/Parsely/streamparse/issues/121 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-704) Apply Travis CI
[ https://issues.apache.org/jira/browse/STORM-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509792#comment-14509792 ] ASF GitHub Bot commented on STORM-704: -- Github user HeartSaVioR commented on a diff in the pull request: https://github.com/apache/storm/pull/486#discussion_r29002804 --- Diff: .travis.yml --- @@ -0,0 +1,7 @@ +language: java +jdk: + - openjdk6 + - openjdk7 --- End diff -- We can start with removing two things now and make new branch which targets two JDKs sometimes. Travis CI will test all PRs and branches, so it would be fine. > Apply Travis CI > --- > > Key: STORM-704 > URL: https://issues.apache.org/jira/browse/STORM-704 > Project: Apache Storm > Issue Type: Improvement > Environment: Travis CI >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Minor > > Now Apache Storm takes advantage of Github, we can apply Travis CI to some > more advantages. > - Build matrix > -- Travis CI supports various JDK versions (openjdk6, openjdk7, oraclejdk7, > oraclejdk8), and it can be tested separately. > - Build automatically > -- pushed new commits, any new PRs > - Integrated with Github > -- Contributors can see his/her PR breaks compilation / test in some minutes. > -- If he/she adds commits to PR, Travis builds it automatically and update > build result. > Please see [https://travis-ci.org/xetorthio/jedis] for example. > There're some hurdles applying Travis CI to Apache Storm project, but we can > overcome these and finally get great CI. > Current hurdles > - asfgit should manage Travis CI setup for the first time > -- other Apache projects already did it by requesting it to INFRA > --- ex. [https://issues.apache.org/jira/browse/INFRA-6161] > - Travis CI restricts stdout with 4M which is too small for Storm maven > output. > -- Change log level in tests to WARN > - In storm-core, we can't see tests failure information on stdout cause it > just prints 'clojure failed'. > -- We need to find a way to upload surefire / clojure tests report files to > somewhere, and uploaded files should be visible easily. > --- Travis supports uploading artifacts to S3 by > [https://github.com/travis-ci/artifacts], but S3 is not free. > -- I'm not familiar with Clojure, but can we print tests summary with clojure > tests as same as Java junit tests? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-704 Apply Travis CI to Apache Storm Proj...
Github user HeartSaVioR commented on a diff in the pull request: https://github.com/apache/storm/pull/486#discussion_r29002804 --- Diff: .travis.yml --- @@ -0,0 +1,7 @@ +language: java +jdk: + - openjdk6 + - openjdk7 --- End diff -- We can start with removing two things now and make new branch which targets two JDKs sometimes. Travis CI will test all PRs and branches, so it would be fine. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-796) The "error" command isn't supported on ShellSpout
[ https://issues.apache.org/jira/browse/STORM-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509788#comment-14509788 ] ASF GitHub Bot commented on STORM-796: -- GitHub user amontalenti opened a pull request: https://github.com/apache/storm/pull/533 Add support for "error" command in ShellSpout Addresses JIRA issue [STORM-796](https://issues.apache.org/jira/browse/STORM-796). The symptoms a multi-lang user will see here is that if their Spout throws an error and their multi-lang implementation sends an "error" command up to the ShellSpout, the ShellSpout will respond saying that it doesn't recognize the "error" command, and thus it will crash (while swallowing the exception thrown by the underlying multi-lang component). ShellBolt already supports the error command, and ShellSpout already has a way of reporting errors -- so this was just a matter of adding similar logic to the ShellSpout. You can merge this pull request into a Git repository by running: $ git pull https://github.com/amontalenti/storm master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/533.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #533 commit 2c35cb4dec67604185d3b443a3e04a66fd4a6a6d Author: Andrew Montalenti Date: 2015-04-23T20:41:39Z add support for "error" command in ShellSpout > The "error" command isn't supported on ShellSpout > - > > Key: STORM-796 > URL: https://issues.apache.org/jira/browse/STORM-796 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 0.9.4 >Reporter: Andrew Montalenti >Assignee: Andrew Montalenti > Fix For: 0.10.0 > > > The `ShellBolt` can handle the "error" command, as shown in this file in > Storm source code: > https://github.com/apache/storm/blob/2dd7a9426e5634211f14cf5c4e10e021d3420c3c/storm-core/src/jvm/backtype/storm/task/ShellBolt.java#L330 > But, `ShellSpout` does not actually have a handler for "error". > https://github.com/apache/storm/blob/2dd7a9426e5634211f14cf5c4e10e021d3420c3c/storm-core/src/jvm/backtype/storm/spout/ShellSpout.java#L153-L175 > The symptoms a multi-lang user will see here is that if their Spout throws an > error and their multi-lang implementation sends an "error" command up to the > ShellSpout, the ShellSpout will respond saying that it doesn't recognize the > "error" command, and thus it will crash (while swallowing the exception > thrown by the underlying multi-lang component). > I am about to open a PR on Github that fixes this. > Originally reported on the streamparse project in this Github issue: > https://github.com/Parsely/streamparse/issues/121 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: Add support for "error" command in ShellSpout
Github user amontalenti commented on the pull request: https://github.com/apache/storm/pull/533#issuecomment-95717037 By the way, I added this and all the existing test cases pass via `mvn`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-704) Apply Travis CI
[ https://issues.apache.org/jira/browse/STORM-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509783#comment-14509783 ] ASF GitHub Bot commented on STORM-704: -- Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/486#issuecomment-95716644 @revans2 About multilang, I missed due to not failing tests. We may like to make multilang tests more reliable. I'll find a reason about what dependencies are missing, and find a way to install such things. About test failures, we can control STORM_TEST_TIMEOUT_MS to respect slow box, but maybe some tests are still needs fast (maybe not very slow) machine. > Apply Travis CI > --- > > Key: STORM-704 > URL: https://issues.apache.org/jira/browse/STORM-704 > Project: Apache Storm > Issue Type: Improvement > Environment: Travis CI >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Minor > > Now Apache Storm takes advantage of Github, we can apply Travis CI to some > more advantages. > - Build matrix > -- Travis CI supports various JDK versions (openjdk6, openjdk7, oraclejdk7, > oraclejdk8), and it can be tested separately. > - Build automatically > -- pushed new commits, any new PRs > - Integrated with Github > -- Contributors can see his/her PR breaks compilation / test in some minutes. > -- If he/she adds commits to PR, Travis builds it automatically and update > build result. > Please see [https://travis-ci.org/xetorthio/jedis] for example. > There're some hurdles applying Travis CI to Apache Storm project, but we can > overcome these and finally get great CI. > Current hurdles > - asfgit should manage Travis CI setup for the first time > -- other Apache projects already did it by requesting it to INFRA > --- ex. [https://issues.apache.org/jira/browse/INFRA-6161] > - Travis CI restricts stdout with 4M which is too small for Storm maven > output. > -- Change log level in tests to WARN > - In storm-core, we can't see tests failure information on stdout cause it > just prints 'clojure failed'. > -- We need to find a way to upload surefire / clojure tests report files to > somewhere, and uploaded files should be visible easily. > --- Travis supports uploading artifacts to S3 by > [https://github.com/travis-ci/artifacts], but S3 is not free. > -- I'm not familiar with Clojure, but can we print tests summary with clojure > tests as same as Java junit tests? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: Add support for "error" command in ShellSpout
GitHub user amontalenti opened a pull request: https://github.com/apache/storm/pull/533 Add support for "error" command in ShellSpout Addresses JIRA issue [STORM-796](https://issues.apache.org/jira/browse/STORM-796). The symptoms a multi-lang user will see here is that if their Spout throws an error and their multi-lang implementation sends an "error" command up to the ShellSpout, the ShellSpout will respond saying that it doesn't recognize the "error" command, and thus it will crash (while swallowing the exception thrown by the underlying multi-lang component). ShellBolt already supports the error command, and ShellSpout already has a way of reporting errors -- so this was just a matter of adding similar logic to the ShellSpout. You can merge this pull request into a Git repository by running: $ git pull https://github.com/amontalenti/storm master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/533.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #533 commit 2c35cb4dec67604185d3b443a3e04a66fd4a6a6d Author: Andrew Montalenti Date: 2015-04-23T20:41:39Z add support for "error" command in ShellSpout --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-704) Apply Travis CI
[ https://issues.apache.org/jira/browse/STORM-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509779#comment-14509779 ] ASF GitHub Bot commented on STORM-704: -- Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/486#discussion_r29002272 --- Diff: .travis.yml --- @@ -0,0 +1,7 @@ +language: java +jdk: + - openjdk6 + - openjdk7 --- End diff -- Both openjdk6 and openjdk7 seem to be failing all the time. https://travis-ci.org/revans2/incubator-storm/jobs/59783142 This looks like a bug in openjdk and glibc combining together. We may not want to test with them if it is not something that we cannot fix. > Apply Travis CI > --- > > Key: STORM-704 > URL: https://issues.apache.org/jira/browse/STORM-704 > Project: Apache Storm > Issue Type: Improvement > Environment: Travis CI >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Minor > > Now Apache Storm takes advantage of Github, we can apply Travis CI to some > more advantages. > - Build matrix > -- Travis CI supports various JDK versions (openjdk6, openjdk7, oraclejdk7, > oraclejdk8), and it can be tested separately. > - Build automatically > -- pushed new commits, any new PRs > - Integrated with Github > -- Contributors can see his/her PR breaks compilation / test in some minutes. > -- If he/she adds commits to PR, Travis builds it automatically and update > build result. > Please see [https://travis-ci.org/xetorthio/jedis] for example. > There're some hurdles applying Travis CI to Apache Storm project, but we can > overcome these and finally get great CI. > Current hurdles > - asfgit should manage Travis CI setup for the first time > -- other Apache projects already did it by requesting it to INFRA > --- ex. [https://issues.apache.org/jira/browse/INFRA-6161] > - Travis CI restricts stdout with 4M which is too small for Storm maven > output. > -- Change log level in tests to WARN > - In storm-core, we can't see tests failure information on stdout cause it > just prints 'clojure failed'. > -- We need to find a way to upload surefire / clojure tests report files to > somewhere, and uploaded files should be visible easily. > --- Travis supports uploading artifacts to S3 by > [https://github.com/travis-ci/artifacts], but S3 is not free. > -- I'm not familiar with Clojure, but can we print tests summary with clojure > tests as same as Java junit tests? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-704 Apply Travis CI to Apache Storm Proj...
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/486#issuecomment-95716644 @revans2 About multilang, I missed due to not failing tests. We may like to make multilang tests more reliable. I'll find a reason about what dependencies are missing, and find a way to install such things. About test failures, we can control STORM_TEST_TIMEOUT_MS to respect slow box, but maybe some tests are still needs fast (maybe not very slow) machine. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-704 Apply Travis CI to Apache Storm Proj...
Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/486#discussion_r29002272 --- Diff: .travis.yml --- @@ -0,0 +1,7 @@ +language: java +jdk: + - openjdk6 + - openjdk7 --- End diff -- Both openjdk6 and openjdk7 seem to be failing all the time. https://travis-ci.org/revans2/incubator-storm/jobs/59783142 This looks like a bug in openjdk and glibc combining together. We may not want to test with them if it is not something that we cannot fix. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (STORM-796) The "error" command isn't supported on ShellSpout
Andrew Montalenti created STORM-796: --- Summary: The "error" command isn't supported on ShellSpout Key: STORM-796 URL: https://issues.apache.org/jira/browse/STORM-796 Project: Apache Storm Issue Type: Bug Affects Versions: 0.9.4 Reporter: Andrew Montalenti Assignee: Andrew Montalenti Fix For: 0.10.0 The `ShellBolt` can handle the "error" command, as shown in this file in Storm source code: https://github.com/apache/storm/blob/2dd7a9426e5634211f14cf5c4e10e021d3420c3c/storm-core/src/jvm/backtype/storm/task/ShellBolt.java#L330 But, `ShellSpout` does not actually have a handler for "error". https://github.com/apache/storm/blob/2dd7a9426e5634211f14cf5c4e10e021d3420c3c/storm-core/src/jvm/backtype/storm/spout/ShellSpout.java#L153-L175 The symptoms a multi-lang user will see here is that if their Spout throws an error and their multi-lang implementation sends an "error" command up to the ShellSpout, the ShellSpout will respond saying that it doesn't recognize the "error" command, and thus it will crash (while swallowing the exception thrown by the underlying multi-lang component). I am about to open a PR on Github that fixes this. Originally reported on the streamparse project in this Github issue: https://github.com/Parsely/streamparse/issues/121 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (STORM-583) Add spout and bolt implementation for Azure Eventhubs
[ https://issues.apache.org/jira/browse/STORM-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509757#comment-14509757 ] P. Taylor Goetz edited comment on STORM-583 at 4/23/15 8:51 PM: IP Clearance has completed: http://mail-archives.apache.org/mod_mbox/incubator-general/201504.mbox/%3cc1f10b96-87c4-4dd8-8433-c4dfa013e...@apache.org%3e was (Author: ptgoetz): IP Clearance has completed: https://issues.apache.org/jira/browse/STORM-583 > Add spout and bolt implementation for Azure Eventhubs > - > > Key: STORM-583 > URL: https://issues.apache.org/jira/browse/STORM-583 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 0.9.3 >Reporter: shanyu zhao >Assignee: shanyu zhao > Fix For: 0.10.0 > > Attachments: Storm-EventHubsDesign.docx, storm-eventhubs.tar.gz > > > Add spout and bolt implementations for Azure Eventhubs - a messaging service > that supports AMQP protocol. Just like storm-kafka/storm-hbase, we need to > add the project to the /external folder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-583) Add spout and bolt implementation for Azure Eventhubs
[ https://issues.apache.org/jira/browse/STORM-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509768#comment-14509768 ] P. Taylor Goetz commented on STORM-583: --- +1 for merging this and I'm willing to act as a committer-sponsor. For future work, I would like to see further documentation regarding deploying/testing, but that can be handled as a separate JIRA. > Add spout and bolt implementation for Azure Eventhubs > - > > Key: STORM-583 > URL: https://issues.apache.org/jira/browse/STORM-583 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 0.9.3 >Reporter: shanyu zhao >Assignee: shanyu zhao > Fix For: 0.10.0 > > Attachments: Storm-EventHubsDesign.docx, storm-eventhubs.tar.gz > > > Add spout and bolt implementations for Azure Eventhubs - a messaging service > that supports AMQP protocol. Just like storm-kafka/storm-hbase, we need to > add the project to the /external folder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-704) Apply Travis CI
[
https://issues.apache.org/jira/browse/STORM-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509767#comment-14509767
]
ASF GitHub Bot commented on STORM-704:
--
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/486#discussion_r29001777
--- Diff: dev-tools/travis/travis-build.sh ---
@@ -0,0 +1,35 @@
+#!/bin/bash
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+STORM_SRC_ROOT_DIR=$1
+
+TRAVIS_SCRIPT_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
+
+cd ${STORM_SRC_ROOT_DIR}
+
+# Travis CI doesn't allow stdout bigger than 4M, so we have to reduce log
while running tests
+export LOG_LEVEL=WARN
+# We should concern that Travis CI could be very slow cause it uses VM
+export export STORM_TEST_TIMEOUT_MS=10
--- End diff --
Thanks for finding! I'll fix it.
> Apply Travis CI
> ---
>
> Key: STORM-704
> URL: https://issues.apache.org/jira/browse/STORM-704
> Project: Apache Storm
> Issue Type: Improvement
> Environment: Travis CI
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>Priority: Minor
>
> Now Apache Storm takes advantage of Github, we can apply Travis CI to some
> more advantages.
> - Build matrix
> -- Travis CI supports various JDK versions (openjdk6, openjdk7, oraclejdk7,
> oraclejdk8), and it can be tested separately.
> - Build automatically
> -- pushed new commits, any new PRs
> - Integrated with Github
> -- Contributors can see his/her PR breaks compilation / test in some minutes.
> -- If he/she adds commits to PR, Travis builds it automatically and update
> build result.
> Please see [https://travis-ci.org/xetorthio/jedis] for example.
> There're some hurdles applying Travis CI to Apache Storm project, but we can
> overcome these and finally get great CI.
> Current hurdles
> - asfgit should manage Travis CI setup for the first time
> -- other Apache projects already did it by requesting it to INFRA
> --- ex. [https://issues.apache.org/jira/browse/INFRA-6161]
> - Travis CI restricts stdout with 4M which is too small for Storm maven
> output.
> -- Change log level in tests to WARN
> - In storm-core, we can't see tests failure information on stdout cause it
> just prints 'clojure failed'.
> -- We need to find a way to upload surefire / clojure tests report files to
> somewhere, and uploaded files should be visible easily.
> --- Travis supports uploading artifacts to S3 by
> [https://github.com/travis-ci/artifacts], but S3 is not free.
> -- I'm not familiar with Clojure, but can we print tests summary with clojure
> tests as same as Java junit tests?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-704 Apply Travis CI to Apache Storm Proj...
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/486#discussion_r29001777
--- Diff: dev-tools/travis/travis-build.sh ---
@@ -0,0 +1,35 @@
+#!/bin/bash
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+STORM_SRC_ROOT_DIR=$1
+
+TRAVIS_SCRIPT_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
+
+cd ${STORM_SRC_ROOT_DIR}
+
+# Travis CI doesn't allow stdout bigger than 4M, so we have to reduce log
while running tests
+export LOG_LEVEL=WARN
+# We should concern that Travis CI could be very slow cause it uses VM
+export export STORM_TEST_TIMEOUT_MS=10
--- End diff --
Thanks for finding! I'll fix it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-704) Apply Travis CI
[
https://issues.apache.org/jira/browse/STORM-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509766#comment-14509766
]
ASF GitHub Bot commented on STORM-704:
--
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/486#discussion_r29001690
--- Diff: dev-tools/travis/travis-build.sh ---
@@ -0,0 +1,35 @@
+#!/bin/bash
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+STORM_SRC_ROOT_DIR=$1
+
+TRAVIS_SCRIPT_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
+
+cd ${STORM_SRC_ROOT_DIR}
+
+# Travis CI doesn't allow stdout bigger than 4M, so we have to reduce log
while running tests
+export LOG_LEVEL=WARN
+# We should concern that Travis CI could be very slow cause it uses VM
+export export STORM_TEST_TIMEOUT_MS=10
+
+mvn clean test
--- End diff --
Yes, it should be changed for now.
I don't want to lean on Travis CI's behavior, but we would love to let test
run as fast as possible, so just run ```mvn test``` with describing would be
great.
> Apply Travis CI
> ---
>
> Key: STORM-704
> URL: https://issues.apache.org/jira/browse/STORM-704
> Project: Apache Storm
> Issue Type: Improvement
> Environment: Travis CI
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>Priority: Minor
>
> Now Apache Storm takes advantage of Github, we can apply Travis CI to some
> more advantages.
> - Build matrix
> -- Travis CI supports various JDK versions (openjdk6, openjdk7, oraclejdk7,
> oraclejdk8), and it can be tested separately.
> - Build automatically
> -- pushed new commits, any new PRs
> - Integrated with Github
> -- Contributors can see his/her PR breaks compilation / test in some minutes.
> -- If he/she adds commits to PR, Travis builds it automatically and update
> build result.
> Please see [https://travis-ci.org/xetorthio/jedis] for example.
> There're some hurdles applying Travis CI to Apache Storm project, but we can
> overcome these and finally get great CI.
> Current hurdles
> - asfgit should manage Travis CI setup for the first time
> -- other Apache projects already did it by requesting it to INFRA
> --- ex. [https://issues.apache.org/jira/browse/INFRA-6161]
> - Travis CI restricts stdout with 4M which is too small for Storm maven
> output.
> -- Change log level in tests to WARN
> - In storm-core, we can't see tests failure information on stdout cause it
> just prints 'clojure failed'.
> -- We need to find a way to upload surefire / clojure tests report files to
> somewhere, and uploaded files should be visible easily.
> --- Travis supports uploading artifacts to S3 by
> [https://github.com/travis-ci/artifacts], but S3 is not free.
> -- I'm not familiar with Clojure, but can we print tests summary with clojure
> tests as same as Java junit tests?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-704 Apply Travis CI to Apache Storm Proj...
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/486#discussion_r29001690
--- Diff: dev-tools/travis/travis-build.sh ---
@@ -0,0 +1,35 @@
+#!/bin/bash
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+STORM_SRC_ROOT_DIR=$1
+
+TRAVIS_SCRIPT_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
+
+cd ${STORM_SRC_ROOT_DIR}
+
+# Travis CI doesn't allow stdout bigger than 4M, so we have to reduce log
while running tests
+export LOG_LEVEL=WARN
+# We should concern that Travis CI could be very slow cause it uses VM
+export export STORM_TEST_TIMEOUT_MS=10
+
+mvn clean test
--- End diff --
Yes, it should be changed for now.
I don't want to lean on Travis CI's behavior, but we would love to let test
run as fast as possible, so just run ```mvn test``` with describing would be
great.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-583) Add spout and bolt implementation for Azure Eventhubs
[ https://issues.apache.org/jira/browse/STORM-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509757#comment-14509757 ] P. Taylor Goetz commented on STORM-583: --- IP Clearance has completed: https://issues.apache.org/jira/browse/STORM-583 > Add spout and bolt implementation for Azure Eventhubs > - > > Key: STORM-583 > URL: https://issues.apache.org/jira/browse/STORM-583 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 0.9.3 >Reporter: shanyu zhao >Assignee: shanyu zhao > Fix For: 0.10.0 > > Attachments: Storm-EventHubsDesign.docx, storm-eventhubs.tar.gz > > > Add spout and bolt implementations for Azure Eventhubs - a messaging service > that supports AMQP protocol. Just like storm-kafka/storm-hbase, we need to > add the project to the /external folder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-704) Apply Travis CI
[ https://issues.apache.org/jira/browse/STORM-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509724#comment-14509724 ] ASF GitHub Bot commented on STORM-704: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/486#issuecomment-95710637 I cloned your changes and have been playing around with them. For me the test all fail before reaching the clojure test. https://travis-ci.org/revans2/incubator-storm/jobs/59781670 I think @kishorvpatil is looking into the test failures. I don't see them on my box, so I think it has something to do with how fast the box is. > Apply Travis CI > --- > > Key: STORM-704 > URL: https://issues.apache.org/jira/browse/STORM-704 > Project: Apache Storm > Issue Type: Improvement > Environment: Travis CI >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Minor > > Now Apache Storm takes advantage of Github, we can apply Travis CI to some > more advantages. > - Build matrix > -- Travis CI supports various JDK versions (openjdk6, openjdk7, oraclejdk7, > oraclejdk8), and it can be tested separately. > - Build automatically > -- pushed new commits, any new PRs > - Integrated with Github > -- Contributors can see his/her PR breaks compilation / test in some minutes. > -- If he/she adds commits to PR, Travis builds it automatically and update > build result. > Please see [https://travis-ci.org/xetorthio/jedis] for example. > There're some hurdles applying Travis CI to Apache Storm project, but we can > overcome these and finally get great CI. > Current hurdles > - asfgit should manage Travis CI setup for the first time > -- other Apache projects already did it by requesting it to INFRA > --- ex. [https://issues.apache.org/jira/browse/INFRA-6161] > - Travis CI restricts stdout with 4M which is too small for Storm maven > output. > -- Change log level in tests to WARN > - In storm-core, we can't see tests failure information on stdout cause it > just prints 'clojure failed'. > -- We need to find a way to upload surefire / clojure tests report files to > somewhere, and uploaded files should be visible easily. > --- Travis supports uploading artifacts to S3 by > [https://github.com/travis-ci/artifacts], but S3 is not free. > -- I'm not familiar with Clojure, but can we print tests summary with clojure > tests as same as Java junit tests? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-704 Apply Travis CI to Apache Storm Proj...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/486#issuecomment-95710637 I cloned your changes and have been playing around with them. For me the test all fail before reaching the clojure test. https://travis-ci.org/revans2/incubator-storm/jobs/59781670 I think @kishorvpatil is looking into the test failures. I don't see them on my box, so I think it has something to do with how fast the box is. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-704) Apply Travis CI
[
https://issues.apache.org/jira/browse/STORM-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509720#comment-14509720
]
ASF GitHub Bot commented on STORM-704:
--
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/486#discussion_r28999818
--- Diff: dev-tools/travis/travis-build.sh ---
@@ -0,0 +1,35 @@
+#!/bin/bash
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+STORM_SRC_ROOT_DIR=$1
+
+TRAVIS_SCRIPT_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
+
+cd ${STORM_SRC_ROOT_DIR}
+
+# Travis CI doesn't allow stdout bigger than 4M, so we have to reduce log
while running tests
+export LOG_LEVEL=WARN
+# We should concern that Travis CI could be very slow cause it uses VM
+export export STORM_TEST_TIMEOUT_MS=10
+
+mvn clean test
--- End diff --
Actually I was wrong. travis-ci runs mvn clean install -DskipTests before
building, so we probably just want to run ```mvn test``` and skip the clean.
> Apply Travis CI
> ---
>
> Key: STORM-704
> URL: https://issues.apache.org/jira/browse/STORM-704
> Project: Apache Storm
> Issue Type: Improvement
> Environment: Travis CI
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>Priority: Minor
>
> Now Apache Storm takes advantage of Github, we can apply Travis CI to some
> more advantages.
> - Build matrix
> -- Travis CI supports various JDK versions (openjdk6, openjdk7, oraclejdk7,
> oraclejdk8), and it can be tested separately.
> - Build automatically
> -- pushed new commits, any new PRs
> - Integrated with Github
> -- Contributors can see his/her PR breaks compilation / test in some minutes.
> -- If he/she adds commits to PR, Travis builds it automatically and update
> build result.
> Please see [https://travis-ci.org/xetorthio/jedis] for example.
> There're some hurdles applying Travis CI to Apache Storm project, but we can
> overcome these and finally get great CI.
> Current hurdles
> - asfgit should manage Travis CI setup for the first time
> -- other Apache projects already did it by requesting it to INFRA
> --- ex. [https://issues.apache.org/jira/browse/INFRA-6161]
> - Travis CI restricts stdout with 4M which is too small for Storm maven
> output.
> -- Change log level in tests to WARN
> - In storm-core, we can't see tests failure information on stdout cause it
> just prints 'clojure failed'.
> -- We need to find a way to upload surefire / clojure tests report files to
> somewhere, and uploaded files should be visible easily.
> --- Travis supports uploading artifacts to S3 by
> [https://github.com/travis-ci/artifacts], but S3 is not free.
> -- I'm not familiar with Clojure, but can we print tests summary with clojure
> tests as same as Java junit tests?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-704 Apply Travis CI to Apache Storm Proj...
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/486#discussion_r28999818
--- Diff: dev-tools/travis/travis-build.sh ---
@@ -0,0 +1,35 @@
+#!/bin/bash
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+STORM_SRC_ROOT_DIR=$1
+
+TRAVIS_SCRIPT_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
+
+cd ${STORM_SRC_ROOT_DIR}
+
+# Travis CI doesn't allow stdout bigger than 4M, so we have to reduce log
while running tests
+export LOG_LEVEL=WARN
+# We should concern that Travis CI could be very slow cause it uses VM
+export export STORM_TEST_TIMEOUT_MS=10
+
+mvn clean test
--- End diff --
Actually I was wrong. travis-ci runs mvn clean install -DskipTests before
building, so we probably just want to run ```mvn test``` and skip the clean.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-704) Apply Travis CI
[
https://issues.apache.org/jira/browse/STORM-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509676#comment-14509676
]
ASF GitHub Bot commented on STORM-704:
--
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/486#discussion_r28996914
--- Diff: dev-tools/travis/travis-build.sh ---
@@ -0,0 +1,35 @@
+#!/bin/bash
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+STORM_SRC_ROOT_DIR=$1
+
+TRAVIS_SCRIPT_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
+
+cd ${STORM_SRC_ROOT_DIR}
+
+# Travis CI doesn't allow stdout bigger than 4M, so we have to reduce log
while running tests
+export LOG_LEVEL=WARN
+# We should concern that Travis CI could be very slow cause it uses VM
+export export STORM_TEST_TIMEOUT_MS=10
--- End diff --
Double export?
> Apply Travis CI
> ---
>
> Key: STORM-704
> URL: https://issues.apache.org/jira/browse/STORM-704
> Project: Apache Storm
> Issue Type: Improvement
> Environment: Travis CI
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>Priority: Minor
>
> Now Apache Storm takes advantage of Github, we can apply Travis CI to some
> more advantages.
> - Build matrix
> -- Travis CI supports various JDK versions (openjdk6, openjdk7, oraclejdk7,
> oraclejdk8), and it can be tested separately.
> - Build automatically
> -- pushed new commits, any new PRs
> - Integrated with Github
> -- Contributors can see his/her PR breaks compilation / test in some minutes.
> -- If he/she adds commits to PR, Travis builds it automatically and update
> build result.
> Please see [https://travis-ci.org/xetorthio/jedis] for example.
> There're some hurdles applying Travis CI to Apache Storm project, but we can
> overcome these and finally get great CI.
> Current hurdles
> - asfgit should manage Travis CI setup for the first time
> -- other Apache projects already did it by requesting it to INFRA
> --- ex. [https://issues.apache.org/jira/browse/INFRA-6161]
> - Travis CI restricts stdout with 4M which is too small for Storm maven
> output.
> -- Change log level in tests to WARN
> - In storm-core, we can't see tests failure information on stdout cause it
> just prints 'clojure failed'.
> -- We need to find a way to upload surefire / clojure tests report files to
> somewhere, and uploaded files should be visible easily.
> --- Travis supports uploading artifacts to S3 by
> [https://github.com/travis-ci/artifacts], but S3 is not free.
> -- I'm not familiar with Clojure, but can we print tests summary with clojure
> tests as same as Java junit tests?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-704 Apply Travis CI to Apache Storm Proj...
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/486#discussion_r28996914
--- Diff: dev-tools/travis/travis-build.sh ---
@@ -0,0 +1,35 @@
+#!/bin/bash
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+STORM_SRC_ROOT_DIR=$1
+
+TRAVIS_SCRIPT_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
+
+cd ${STORM_SRC_ROOT_DIR}
+
+# Travis CI doesn't allow stdout bigger than 4M, so we have to reduce log
while running tests
+export LOG_LEVEL=WARN
+# We should concern that Travis CI could be very slow cause it uses VM
+export export STORM_TEST_TIMEOUT_MS=10
--- End diff --
Double export?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-704) Apply Travis CI
[
https://issues.apache.org/jira/browse/STORM-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509647#comment-14509647
]
ASF GitHub Bot commented on STORM-704:
--
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/486#discussion_r28995728
--- Diff: dev-tools/travis/travis-build.sh ---
@@ -0,0 +1,35 @@
+#!/bin/bash
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+STORM_SRC_ROOT_DIR=$1
+
+TRAVIS_SCRIPT_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
+
+cd ${STORM_SRC_ROOT_DIR}
+
+# Travis CI doesn't allow stdout bigger than 4M, so we have to reduce log
while running tests
+export LOG_LEVEL=WARN
+# We should concern that Travis CI could be very slow cause it uses VM
+export export STORM_TEST_TIMEOUT_MS=10
+
+mvn clean test
--- End diff --
After some new changes went in around packaging shell dependencies this
need to be changed to
```mvn clean package```
or maven gets angry about unpacking something that was not already packaged
> Apply Travis CI
> ---
>
> Key: STORM-704
> URL: https://issues.apache.org/jira/browse/STORM-704
> Project: Apache Storm
> Issue Type: Improvement
> Environment: Travis CI
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>Priority: Minor
>
> Now Apache Storm takes advantage of Github, we can apply Travis CI to some
> more advantages.
> - Build matrix
> -- Travis CI supports various JDK versions (openjdk6, openjdk7, oraclejdk7,
> oraclejdk8), and it can be tested separately.
> - Build automatically
> -- pushed new commits, any new PRs
> - Integrated with Github
> -- Contributors can see his/her PR breaks compilation / test in some minutes.
> -- If he/she adds commits to PR, Travis builds it automatically and update
> build result.
> Please see [https://travis-ci.org/xetorthio/jedis] for example.
> There're some hurdles applying Travis CI to Apache Storm project, but we can
> overcome these and finally get great CI.
> Current hurdles
> - asfgit should manage Travis CI setup for the first time
> -- other Apache projects already did it by requesting it to INFRA
> --- ex. [https://issues.apache.org/jira/browse/INFRA-6161]
> - Travis CI restricts stdout with 4M which is too small for Storm maven
> output.
> -- Change log level in tests to WARN
> - In storm-core, we can't see tests failure information on stdout cause it
> just prints 'clojure failed'.
> -- We need to find a way to upload surefire / clojure tests report files to
> somewhere, and uploaded files should be visible easily.
> --- Travis supports uploading artifacts to S3 by
> [https://github.com/travis-ci/artifacts], but S3 is not free.
> -- I'm not familiar with Clojure, but can we print tests summary with clojure
> tests as same as Java junit tests?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-704 Apply Travis CI to Apache Storm Proj...
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/486#discussion_r28995728
--- Diff: dev-tools/travis/travis-build.sh ---
@@ -0,0 +1,35 @@
+#!/bin/bash
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+STORM_SRC_ROOT_DIR=$1
+
+TRAVIS_SCRIPT_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
+
+cd ${STORM_SRC_ROOT_DIR}
+
+# Travis CI doesn't allow stdout bigger than 4M, so we have to reduce log
while running tests
+export LOG_LEVEL=WARN
+# We should concern that Travis CI could be very slow cause it uses VM
+export export STORM_TEST_TIMEOUT_MS=10
+
+mvn clean test
--- End diff --
After some new changes went in around packaging shell dependencies this
need to be changed to
```mvn clean package```
or maven gets angry about unpacking something that was not already packaged
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
Re: Wish to hear your opinion on adopting public CI
I've created an issue with the Apache Infrastructure guys for enabling Travis CI on our git repo. https://issues.apache.org/jira/browse/INFRA-9511 -- Kyle On Thursday, March 26, 2015 5:23 PM, 임정택 wrote: Yes, we should ask Apache Infra to set up Travis CI (with asfgit account), which is not an issue cause many projects on Apache already uses Travis CI. ex. https://issues.apache.org/jira/browse/INFRA-6161 I've just finished work about adopting Travis CI (with my fork), so we're ready! https://travis-ci.org/HeartSaVioR/storm/builds/56021889 Could any committers help requesting set up Travis CI to Apache Infra? Btw, jira issue link is here, https://issues.apache.org/jira/browse/STORM-704 and PR is here, https://github.com/apache/storm/pull/486 Thanks in advance! ps. Please note that Travis CI becomes too slow, and it actually runs mvn install -DskipTests before build, so it takes some mins to assign VM and boot, and another 10 ~ 15 mins to build for now. 2015-03-26 23:54 GMT+09:00 Bobby Evans : > I would love to see CI setup. I don't know how Apache feels about > travis-ci, we mostly ran into issues with setting up ruby on the jenkins > nodes. Now I am sure there will be issues with nodejs. If travis-ci makes > this simpler I would be all for using it. But because it requires some > github hooks we will need Apache Infra to help out setting it up. > - Bobby > > > > On Wednesday, March 25, 2015 9:39 PM, 임정택 wrote: > > > Parth, > Thanks for information! > Since there were some trials, my trial seems to be valid. :) > > Btw, I've seen succeed build from Travis CI, please see > https://travis-ci.org/HeartSaVioR/storm/jobs/54211636 > Based on this we can see we already overcome multi-lang test, too. > > I've got another hurdle, we should open test-reports directory to see > clojure tests, which I can't find clearer solution. > We should rely on stdout / stderr, or upload these to somewhere. > > Please expose your any ideations about this. > > Regards. > Jungtaek Lim (HeartSaVioR) > > 2015-03-26 9:51 GMT+09:00 Parth Brahmbhatt : > > > IIRC, Taylor and Bobby both tried to setup CI and got stuck because of > > multi-lang feature which needs ruby/node.js/python to be installed on the > > host. > > > > Thanks > > Parth > > > > On 3/25/15, 5:10 PM, "임정택" wrote: > > > > >Hi all! > > > > > >I'd like to hear your opinion on adopting public CI. > > >Now Apache Storm takes advantage of Github, we can apply Travis CI to > some > > >more advantages. > > >Detailed informations are on JIRA issue ( > > >https://issues.apache.org/jira/browse/STORM-704). > > > > > >If you don't know Travis CI, it's better to take a look at. > > >https://travis-ci.org/xetorthio/jedis > > > > > >Actually I'm trying to integrate it, and resolved many hurdles but got > > >stuck. > > > > > >If you're interested in, please participate to resolve issue. :) > > > > > >Important question on this, I don't know why Storm didn't have CI. > > >Is there some kind of decision about not having CI? > > >If then this issue should be resolved as invalid. > > > > > >Thanks! > > > > > >Regards. > > >Jungtaek Lim (HeartSaVioR) > > > > > > > -- > Name : 임 정택 > Blog : http://www.heartsavior.net / http://dev.heartsavior.net > Twitter : http://twitter.com/heartsavior > LinkedIn : http://www.linkedin.com/in/heartsavior > > > -- Name : 임 정택 Blog : http://www.heartsavior.net / http://dev.heartsavior.net Twitter : http://twitter.com/heartsavior LinkedIn : http://www.linkedin.com/in/heartsavior
[jira] [Commented] (STORM-704) Apply Travis CI
[ https://issues.apache.org/jira/browse/STORM-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509626#comment-14509626 ] ASF GitHub Bot commented on STORM-704: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/486#issuecomment-95691020 @HeartSaVioR Sorry it has taken me so long to take a look. It appears that for https://travis-ci.org/HeartSaVioR/storm/jobs/56016756 you are missing one of the shell bolt/spout dependencies. We currently need ruby, python, and nodejs installed. The errors about ```39877 [Thread-128] ERROR backtype.storm.task.ShellBolt - Halting process: ShellBolt died. java.lang.RuntimeException: backtype.storm.multilang.NoOutputException: Pipe to subprocess seems to be broken! No output read.``` indicate that one of these is not installed/on the path. > Apply Travis CI > --- > > Key: STORM-704 > URL: https://issues.apache.org/jira/browse/STORM-704 > Project: Apache Storm > Issue Type: Improvement > Environment: Travis CI >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Minor > > Now Apache Storm takes advantage of Github, we can apply Travis CI to some > more advantages. > - Build matrix > -- Travis CI supports various JDK versions (openjdk6, openjdk7, oraclejdk7, > oraclejdk8), and it can be tested separately. > - Build automatically > -- pushed new commits, any new PRs > - Integrated with Github > -- Contributors can see his/her PR breaks compilation / test in some minutes. > -- If he/she adds commits to PR, Travis builds it automatically and update > build result. > Please see [https://travis-ci.org/xetorthio/jedis] for example. > There're some hurdles applying Travis CI to Apache Storm project, but we can > overcome these and finally get great CI. > Current hurdles > - asfgit should manage Travis CI setup for the first time > -- other Apache projects already did it by requesting it to INFRA > --- ex. [https://issues.apache.org/jira/browse/INFRA-6161] > - Travis CI restricts stdout with 4M which is too small for Storm maven > output. > -- Change log level in tests to WARN > - In storm-core, we can't see tests failure information on stdout cause it > just prints 'clojure failed'. > -- We need to find a way to upload surefire / clojure tests report files to > somewhere, and uploaded files should be visible easily. > --- Travis supports uploading artifacts to S3 by > [https://github.com/travis-ci/artifacts], but S3 is not free. > -- I'm not familiar with Clojure, but can we print tests summary with clojure > tests as same as Java junit tests? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-704 Apply Travis CI to Apache Storm Proj...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/486#issuecomment-95691020 @HeartSaVioR Sorry it has taken me so long to take a look. It appears that for https://travis-ci.org/HeartSaVioR/storm/jobs/56016756 you are missing one of the shell bolt/spout dependencies. We currently need ruby, python, and nodejs installed. The errors about ```39877 [Thread-128] ERROR backtype.storm.task.ShellBolt - Halting process: ShellBolt died. java.lang.RuntimeException: backtype.storm.multilang.NoOutputException: Pipe to subprocess seems to be broken! No output read.``` indicate that one of these is not installed/on the path. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-735] [storm-redis] Upgrade Jedis to 2.7...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/491 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (STORM-730) UI: Component Page has extra } for executor ids of Bolts
[ https://issues.apache.org/jira/browse/STORM-730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-730. --- Resolution: Fixed Fix Version/s: 0.11.0 Thanks [~dagit], I merged this into master. > UI: Component Page has extra } for executor ids of Bolts > > > Key: STORM-730 > URL: https://issues.apache.org/jira/browse/STORM-730 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 0.10.0 >Reporter: Derek Dagit >Assignee: Derek Dagit >Priority: Minor > Fix For: 0.11.0 > > > When the component is a Bolt, the Executors table Id values show an extra > closing curly brace. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-730) UI: Component Page has extra } for executor ids of Bolts
[ https://issues.apache.org/jira/browse/STORM-730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509615#comment-14509615 ] ASF GitHub Bot commented on STORM-730: -- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/487 > UI: Component Page has extra } for executor ids of Bolts > > > Key: STORM-730 > URL: https://issues.apache.org/jira/browse/STORM-730 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 0.10.0 >Reporter: Derek Dagit >Assignee: Derek Dagit >Priority: Minor > Fix For: 0.11.0 > > > When the component is a Bolt, the Executors table Id values show an extra > closing curly brace. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-730] remove extra curly brace
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/487 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-735) [storm-redis] Upgrade Jedis to 2.7.0
[ https://issues.apache.org/jira/browse/STORM-735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509611#comment-14509611 ] ASF GitHub Bot commented on STORM-735: -- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/491 > [storm-redis] Upgrade Jedis to 2.7.0 > > > Key: STORM-735 > URL: https://issues.apache.org/jira/browse/STORM-735 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 0.10.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Minor > > Jedis 2.6.2 and 2.7.0 released just now. > It contains some bug fixes, applies tiny optimization, and adds some commands > which are missing. > * 2.6.3: > https://github.com/xetorthio/jedis/issues?q=milestone%3A2.6.3+is%3Aclosed > * 2.7.0: > https://github.com/xetorthio/jedis/issues?q=milestone%3A2.7.0+is%3Aclosed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-730) UI: Component Page has extra } for executor ids of Bolts
[ https://issues.apache.org/jira/browse/STORM-730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509614#comment-14509614 ] ASF GitHub Bot commented on STORM-730: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/487#issuecomment-95688968 +1 > UI: Component Page has extra } for executor ids of Bolts > > > Key: STORM-730 > URL: https://issues.apache.org/jira/browse/STORM-730 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 0.10.0 >Reporter: Derek Dagit >Assignee: Derek Dagit >Priority: Minor > > When the component is a Bolt, the Executors table Id values show an extra > closing curly brace. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-735) [storm-redis] Upgrade Jedis to 2.7.0
[ https://issues.apache.org/jira/browse/STORM-735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-735. --- Resolution: Fixed Fix Version/s: 0.11.0 Thanks [~kabhwan], I merged this into master. > [storm-redis] Upgrade Jedis to 2.7.0 > > > Key: STORM-735 > URL: https://issues.apache.org/jira/browse/STORM-735 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 0.10.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Minor > Fix For: 0.11.0 > > > Jedis 2.6.2 and 2.7.0 released just now. > It contains some bug fixes, applies tiny optimization, and adds some commands > which are missing. > * 2.6.3: > https://github.com/xetorthio/jedis/issues?q=milestone%3A2.6.3+is%3Aclosed > * 2.7.0: > https://github.com/xetorthio/jedis/issues?q=milestone%3A2.7.0+is%3Aclosed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-735] [storm-redis] Upgrade Jedis to 2.7...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/491#issuecomment-95688610 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-730] remove extra curly brace
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/487#issuecomment-95688968 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-735) [storm-redis] Upgrade Jedis to 2.7.0
[ https://issues.apache.org/jira/browse/STORM-735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509610#comment-14509610 ] ASF GitHub Bot commented on STORM-735: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/491#issuecomment-95688610 +1 > [storm-redis] Upgrade Jedis to 2.7.0 > > > Key: STORM-735 > URL: https://issues.apache.org/jira/browse/STORM-735 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 0.10.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Minor > > Jedis 2.6.2 and 2.7.0 released just now. > It contains some bug fixes, applies tiny optimization, and adds some commands > which are missing. > * 2.6.3: > https://github.com/xetorthio/jedis/issues?q=milestone%3A2.6.3+is%3Aclosed > * 2.7.0: > https://github.com/xetorthio/jedis/issues?q=milestone%3A2.7.0+is%3Aclosed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: stormjar.jar on classpath first in supervisor ...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/494#issuecomment-95684780 The problem here is one of classpath hell. The reason someone wants their jar first in the classpath is because there is a conflict between something that storm provides and something that the worker wants. In some cases the developers of the dependency in question have maintained binary backwards compatibility and this will work fine, but in too many cases, i.e. guava, compatibility is not maintained. If the incompatibility results in a method not found error or something like that it is simple to debug, but I have seen issues, like with disruptor, where the errors are a lot more subtle, and everything mostly works. Debugging those over a mailing list when I don't really know what dependencies storm is running against is a real pain. I am fine with having the option to make the user jar first on the classpath, but I would want it to be off by default. @emidln if you want to do that please file a JIRA for it at https://issues.apache.org/jira under STORM and then update the title of the pull request to include the JIRA number, i.e. `STORM-: optionally let stormjar.jar fist on classpath` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (STORM-747) assignment-version-callback/info-with-version-callback are not fired when assignments change
[
https://issues.apache.org/jira/browse/STORM-747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans resolved STORM-747.
---
Resolution: Fixed
Fix Version/s: 0.11.0
Thanks [~mansheng],
I merged this into master.
> assignment-version-callback/info-with-version-callback are not fired when
> assignments change
>
>
> Key: STORM-747
> URL: https://issues.apache.org/jira/browse/STORM-747
> Project: Apache Storm
> Issue Type: Bug
>Reporter: Mansheng Yang
>Assignee: Mansheng Yang
>Priority: Minor
> Fix For: 0.11.0
>
>
> In cluster.clj, assignment-version-callback and
> assignment-info-with-version-callback are never fired when assignments
> change. These callbacks are used in the worker and expected to be called when
> the corresponding assignment changes.
> Starting from cluster.clj line 269:
> {code}
> (fn [type path]
> (let [[subtree & args] (tokenize-path path)]
> (condp = subtree
> ASSIGNMENTS-ROOT (if (empty? args)
> (issue-callback!
> assignments-callback)
> (issue-map-callback!
> assignment-info-callback (first args)))
> SUPERVISORS-ROOT (issue-callback!
> supervisors-callback)
> STORMS-ROOT (issue-map-callback! storm-base-callback
> (first args))
> CREDENTIALS-ROOT (issue-map-callback!
> credentials-callback (first args))
> {code}
> It is clear that only assignment-info-callback is fired
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-747) assignment-version-callback/info-with-version-callback are not fired when assignments change
[
https://issues.apache.org/jira/browse/STORM-747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509312#comment-14509312
]
ASF GitHub Bot commented on STORM-747:
--
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/500
> assignment-version-callback/info-with-version-callback are not fired when
> assignments change
>
>
> Key: STORM-747
> URL: https://issues.apache.org/jira/browse/STORM-747
> Project: Apache Storm
> Issue Type: Bug
>Reporter: Mansheng Yang
>Assignee: Mansheng Yang
>Priority: Minor
>
> In cluster.clj, assignment-version-callback and
> assignment-info-with-version-callback are never fired when assignments
> change. These callbacks are used in the worker and expected to be called when
> the corresponding assignment changes.
> Starting from cluster.clj line 269:
> {code}
> (fn [type path]
> (let [[subtree & args] (tokenize-path path)]
> (condp = subtree
> ASSIGNMENTS-ROOT (if (empty? args)
> (issue-callback!
> assignments-callback)
> (issue-map-callback!
> assignment-info-callback (first args)))
> SUPERVISORS-ROOT (issue-callback!
> supervisors-callback)
> STORMS-ROOT (issue-map-callback! storm-base-callback
> (first args))
> CREDENTIALS-ROOT (issue-map-callback!
> credentials-callback (first args))
> {code}
> It is clear that only assignment-info-callback is fired
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: Issue assignment-version/info-with-version-cal...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/500 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Issue assignment-version/info-with-version-cal...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/500#issuecomment-95648046 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: Why is Storm Config not modifiable?
Yes you could all assoc on the Map, if you cast the map to an IPersistentMap and call the assoc method on it, but if you do it blindly without checking they actual type of the Map passed in you may run into issues in the future, if we do change some internals away from java. - Bobby On Thursday, April 23, 2015 9:07 AM, Jeremy Heiler wrote: On Thu, Apr 23, 2015 at 4:42 AM, Matthias J. Sax < mj...@informatik.hu-berlin.de> wrote: > In my case, I use a class hierarchy with abstract spouts/bolts and > multiple concrete spouts/bolts. The abstract class uses some default > values (if not provided in the given config) similar to the derived > classes. However, some derived classes overwrite the default values for > the abstract class, ie, if no value is given in the config, some derived > classes set the values and other do not. Additionally, open()/prepare() > of the abstract class is called, too, and set the default value if the > value was neither provided by the user, not by the derived class. > > The idiomatic way to update an immutable field like this in Clojure would be to use an atom. Here you could store the conf in an AtomicReference field, call .assoc on the map to update it, and .compareAndSet it on the AtomicReference. I'm not sure you'll need this, but I figured I'd mention it as how you would use a PersistentHashMap in your situation.
Re: Why is Storm Config not modifiable?
On Thu, Apr 23, 2015 at 4:42 AM, Matthias J. Sax < mj...@informatik.hu-berlin.de> wrote: > In my case, I use a class hierarchy with abstract spouts/bolts and > multiple concrete spouts/bolts. The abstract class uses some default > values (if not provided in the given config) similar to the derived > classes. However, some derived classes overwrite the default values for > the abstract class, ie, if no value is given in the config, some derived > classes set the values and other do not. Additionally, open()/prepare() > of the abstract class is called, too, and set the default value if the > value was neither provided by the user, not by the derived class. > > The idiomatic way to update an immutable field like this in Clojure would be to use an atom. Here you could store the conf in an AtomicReference field, call .assoc on the map to update it, and .compareAndSet it on the AtomicReference. I'm not sure you'll need this, but I figured I'd mention it as how you would use a PersistentHashMap in your situation.
[jira] [Commented] (STORM-790) Log "task id is null" instead of let worker died (NPE in consumeBatchToCursor)
[ https://issues.apache.org/jira/browse/STORM-790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509096#comment-14509096 ] ASF GitHub Bot commented on STORM-790: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/527#issuecomment-95593966 @HeartSaVioR great I am still +1 on the pull request. > Log "task id is null" instead of let worker died (NPE in consumeBatchToCursor) > -- > > Key: STORM-790 > URL: https://issues.apache.org/jira/browse/STORM-790 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 0.9.2-incubating, 0.9.3, 0.10.0, 0.9.4, 0.11.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim > > In STORM-770, some users have observed that worker suddenly died with NPE in > consumeBatchToCursor(). > Looks like it can occur when "task" in "mk-transfer-fn" is null. > It may not be an issue before 0.9.2-incubating, since worker just ignores > that tuple. > Please see > https://issues.apache.org/jira/browse/STORM-770?focusedCommentId=14496199&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14496199. > Before finding root cause of this issue, it would be better to let worker not > killed by this issue but just log with WARN or ERROR level. > It really makes sense cause before 0.9.2 Storm silently ignores tuple, and > with Guaranteeing Message Processing, after timed-out tuple will be replayed. > (It isn't applied to non-ack) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-790 Log "task is null" instead of let wo...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/527#issuecomment-95593966 @HeartSaVioR great I am still +1 on the pull request. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: Why is Storm Config not modifiable?
The persistent map comes from clojure. Clojure is a functional language where
all data types are immutable. My suggestion would be to keep doing what you
are doing.
open(..., Map conf, ...) { conf = new HashMap(conf); ...
}
I know it can be a pain, but it also avoids situations like with Hadoop, where
configs are modifiable and shared. I have had to fix race conditions because
one part of the code is setting a config that it thinks no one else is using
right now but another part of the code on a different thread is reading that
same config, and sometimes it reads the old one, and other times it reads the
new one. I really would rather have had an UnsupportedOperationException come
out early on instead of trying to debug what was happening.
You could also create a few abstract parent classes that all of your bolts and
spouts derive from.
... open(..., Map conf, ...) { modOpen(...,conf instanceof HashMap ?
(HashMap)conf : new HashMap(conf), ...);
}
... abstract modOpen(..., HashMap conf, ...);
- Bobby
On Thursday, April 23, 2015 8:10 AM, Nathan Leung
wrote:
Instead of writing the map you can just write a variable in the abstract
class.
On Apr 23, 2015 7:45 AM, "Matthias J. Sax"
wrote:
> I understand that.
>
> In my case, I use a class hierarchy with abstract spouts/bolts and
> multiple concrete spouts/bolts. The abstract class uses some default
> values (if not provided in the given config) similar to the derived
> classes. However, some derived classes overwrite the default values for
> the abstract class, ie, if no value is given in the config, some derived
> classes set the values and other do not. Additionally, open()/prepare()
> of the abstract class is called, too, and set the default value if the
> value was neither provided by the user, not by the derived class.
>
> Does it make sense how I explained it?
>
>
> -Matthias
>
>
>
>
> On 04/23/2015 01:08 PM, Nathan Leung wrote:
> > Config should be set when the topology is created before it's sent to the
> > nimbus. If you write an option to the map there is no mechanism to
> > propagate the change to the rest of the instances of storm components.
> > On Apr 23, 2015 6:02 AM, "Matthias J. Sax" <
> mj...@informatik.hu-berlin.de>
> > wrote:
> >
> >> Hi,
> >>
> >> in the Spout/Bolt open()/prepare() method, a Map containing the current
> >> configuration is given. This Map is of type
> >> clojure.lang.PersistentHashMap and calling .put(...) raises an
> >> UnsupportedOperationException.
> >>
> >> I was wondering, why a persistent map is used? I could image, that
> >> system defined values should not be modified. But adding new key/value
> >> pairs is not allowed either (and would be very helpful in my case).
> >>
> >> Right now, I simply copy all values into a new HashMap object and add my
> >> own values to this new HashMap. But this is an annoying workaround I
> >> have to do each time...
> >>
> >> Would it make sense, the change the type of the system provided Map (or
> >> modify it's behavior to all for adding new key/value pairs)?
> >>
> >>
> >> -Matthias
> >>
> >>
> >>
> >>
> >>
> >
>
>
[jira] [Comment Edited] (STORM-770) NullPointerException in consumeBatchToCursor
[
https://issues.apache.org/jira/browse/STORM-770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509038#comment-14509038
]
Michael Pershyn edited comment on STORM-770 at 4/23/15 1:28 PM:
So, I have got the first one. Again, it happens after the bolt was initialized
after rebalance (not on initial submit).
This is the line that is built into the storm-core (from patch):
{code}
(log-warn "Can't transfer tuple - task value is null. tuple type: " (type
tuple) " and information: " tuple))
{code}
This is the result that it produces:
{code}
// happens strait after {{prepare}} method of a bolt.
2015-04-23T14:46:59.517+0200 b.s.d.worker [WARN] Can't transfer tuple - task
value is null. tuple type: and information:
{code}
I am confused why we don't see {{nil}} there. Is it possible that {{log-warn}}
prints nothing instead?
was (Author: pershyn):
So, I have got the first one. Again. happens after the bolt was initialized
after rebalance (not on initial submit).
This is the line that is built into the storm-core (from patch):
{code}
(log-warn "Can't transfer tuple - task value is null. tuple type: " (type
tuple) " and information: " tuple))
{code}
This is the result that it produces:
{code}
// happens strait after {{prepare}} method of a bolt.
2015-04-23T14:46:59.517+0200 b.s.d.worker [WARN] Can't transfer tuple - task
value is null. tuple type: and information:
{code}
I am confused why we don't see 'nil' there. Is it possible that {{log-warn}}
prints nothing instead?
> NullPointerException in consumeBatchToCursor
>
>
> Key: STORM-770
> URL: https://issues.apache.org/jira/browse/STORM-770
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 0.9.2-incubating
>Reporter: Stas Levin
>
> We got the following exception after our topology had been up for ~2 days,
> and I was wondering if it might be related.
> Looks like "task" in "mk-transfer-fn" is null, making "(.add remote
> (TaskMessage. task (.serialize serializer tuple)))" fail on NPE
> (worker.clj:128, storm-core-0.9.2-incubating.jar)
> java.lang.RuntimeException: java.lang.NullPointerException
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$consume_loop_STAR_$fn__758.invoke(disruptor.clj:94)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_72]
> Caused by: java.lang.NullPointerException: null
> at clojure.lang.RT.intCast(RT.java:1087) ~[clojure-1.5.1.jar:na]
> at
> backtype.storm.daemon.worker$mk_transfer_fn$fn__5748.invoke(worker.clj:128)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.executor$start_batch_transfer_GT_worker_handler_BANG$fn__5483.invoke(executor.clj:256)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> ... 6 common frames omitted,java.lang.RuntimeException:
> java.lang.NullPointerException
> Any ideas?
> P.S.
> Also saw it here:
> http://mail-archives.apache.org/mod_mbox/storm-user/201501.mbox/%3CCABcMBhCusXXU=v1e66wfuatgyh1euqnd1siog65-tp8xlwx...@mail.gmail.com%3E
> https://mail-archives.apache.org/mod_mbox/storm-user/201408.mbox/%3ccajuqm_4kxhsh2_x08ujuqr76m2c+dswp0fcijbmfcaeyqgs...@mail.gmail.com%3E
> Comment from Bobby
> http://mail-archives.apache.org/mod_mbox/storm-user/201501.mbox/%3c574363643.2791948.1420470097280.javamail.ya...@jws10027.mail.ne1.yahoo.com%3E
> {quote}
> What version of storm are you using? Are any of the bolts shell bolts?
> There is a known
> issue where this can happen if two shell bolts share an executor, because
> they are multi-threaded.
> - Bobby
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-770) NullPointerException in consumeBatchToCursor
[
https://issues.apache.org/jira/browse/STORM-770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509038#comment-14509038
]
Michael Pershyn commented on STORM-770:
---
So, I have got the first one. Again. happens after the bolt was initialized
after rebalance (not on initial submit).
This is the line that is built into the storm-core (from patch):
{code}
(log-warn "Can't transfer tuple - task value is null. tuple type: " (type
tuple) " and information: " tuple))
{code}
This is the result that it produces:
{code}
// happens strait after {{prepare}} method of a bolt.
2015-04-23T14:46:59.517+0200 b.s.d.worker [WARN] Can't transfer tuple - task
value is null. tuple type: and information:
{code}
I am confused why we don't see 'nil' there. Is it possible that {{log-warn}}
prints nothing instead?
> NullPointerException in consumeBatchToCursor
>
>
> Key: STORM-770
> URL: https://issues.apache.org/jira/browse/STORM-770
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 0.9.2-incubating
>Reporter: Stas Levin
>
> We got the following exception after our topology had been up for ~2 days,
> and I was wondering if it might be related.
> Looks like "task" in "mk-transfer-fn" is null, making "(.add remote
> (TaskMessage. task (.serialize serializer tuple)))" fail on NPE
> (worker.clj:128, storm-core-0.9.2-incubating.jar)
> java.lang.RuntimeException: java.lang.NullPointerException
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$consume_loop_STAR_$fn__758.invoke(disruptor.clj:94)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_72]
> Caused by: java.lang.NullPointerException: null
> at clojure.lang.RT.intCast(RT.java:1087) ~[clojure-1.5.1.jar:na]
> at
> backtype.storm.daemon.worker$mk_transfer_fn$fn__5748.invoke(worker.clj:128)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.executor$start_batch_transfer_GT_worker_handler_BANG$fn__5483.invoke(executor.clj:256)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> ... 6 common frames omitted,java.lang.RuntimeException:
> java.lang.NullPointerException
> Any ideas?
> P.S.
> Also saw it here:
> http://mail-archives.apache.org/mod_mbox/storm-user/201501.mbox/%3CCABcMBhCusXXU=v1e66wfuatgyh1euqnd1siog65-tp8xlwx...@mail.gmail.com%3E
> https://mail-archives.apache.org/mod_mbox/storm-user/201408.mbox/%3ccajuqm_4kxhsh2_x08ujuqr76m2c+dswp0fcijbmfcaeyqgs...@mail.gmail.com%3E
> Comment from Bobby
> http://mail-archives.apache.org/mod_mbox/storm-user/201501.mbox/%3c574363643.2791948.1420470097280.javamail.ya...@jws10027.mail.ne1.yahoo.com%3E
> {quote}
> What version of storm are you using? Are any of the bolts shell bolts?
> There is a known
> issue where this can happen if two shell bolts share an executor, because
> they are multi-threaded.
> - Bobby
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Why is Storm Config not modifiable?
Instead of writing the map you can just write a variable in the abstract class. On Apr 23, 2015 7:45 AM, "Matthias J. Sax" wrote: > I understand that. > > In my case, I use a class hierarchy with abstract spouts/bolts and > multiple concrete spouts/bolts. The abstract class uses some default > values (if not provided in the given config) similar to the derived > classes. However, some derived classes overwrite the default values for > the abstract class, ie, if no value is given in the config, some derived > classes set the values and other do not. Additionally, open()/prepare() > of the abstract class is called, too, and set the default value if the > value was neither provided by the user, not by the derived class. > > Does it make sense how I explained it? > > > -Matthias > > > > > On 04/23/2015 01:08 PM, Nathan Leung wrote: > > Config should be set when the topology is created before it's sent to the > > nimbus. If you write an option to the map there is no mechanism to > > propagate the change to the rest of the instances of storm components. > > On Apr 23, 2015 6:02 AM, "Matthias J. Sax" < > mj...@informatik.hu-berlin.de> > > wrote: > > > >> Hi, > >> > >> in the Spout/Bolt open()/prepare() method, a Map containing the current > >> configuration is given. This Map is of type > >> clojure.lang.PersistentHashMap and calling .put(...) raises an > >> UnsupportedOperationException. > >> > >> I was wondering, why a persistent map is used? I could image, that > >> system defined values should not be modified. But adding new key/value > >> pairs is not allowed either (and would be very helpful in my case). > >> > >> Right now, I simply copy all values into a new HashMap object and add my > >> own values to this new HashMap. But this is an annoying workaround I > >> have to do each time... > >> > >> Would it make sense, the change the type of the system provided Map (or > >> modify it's behavior to all for adding new key/value pairs)? > >> > >> > >> -Matthias > >> > >> > >> > >> > >> > > > >
[jira] [Comment Edited] (STORM-770) NullPointerException in consumeBatchToCursor
[
https://issues.apache.org/jira/browse/STORM-770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14508987#comment-14508987
]
Jungtaek Lim edited comment on STORM-770 at 4/23/15 12:46 PM:
--
Great!
I'm also looking forward to see it. Thanks for follow up again! :)
Btw, it'll print type / information of both tuple (which was a reason to crash)
and whole tuple-batch, right?
was (Author: kabhwan):
Great!
I'm also looking forward to see it. Thanks for follow up again! :)
> NullPointerException in consumeBatchToCursor
>
>
> Key: STORM-770
> URL: https://issues.apache.org/jira/browse/STORM-770
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 0.9.2-incubating
>Reporter: Stas Levin
>
> We got the following exception after our topology had been up for ~2 days,
> and I was wondering if it might be related.
> Looks like "task" in "mk-transfer-fn" is null, making "(.add remote
> (TaskMessage. task (.serialize serializer tuple)))" fail on NPE
> (worker.clj:128, storm-core-0.9.2-incubating.jar)
> java.lang.RuntimeException: java.lang.NullPointerException
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$consume_loop_STAR_$fn__758.invoke(disruptor.clj:94)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_72]
> Caused by: java.lang.NullPointerException: null
> at clojure.lang.RT.intCast(RT.java:1087) ~[clojure-1.5.1.jar:na]
> at
> backtype.storm.daemon.worker$mk_transfer_fn$fn__5748.invoke(worker.clj:128)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.executor$start_batch_transfer_GT_worker_handler_BANG$fn__5483.invoke(executor.clj:256)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> ... 6 common frames omitted,java.lang.RuntimeException:
> java.lang.NullPointerException
> Any ideas?
> P.S.
> Also saw it here:
> http://mail-archives.apache.org/mod_mbox/storm-user/201501.mbox/%3CCABcMBhCusXXU=v1e66wfuatgyh1euqnd1siog65-tp8xlwx...@mail.gmail.com%3E
> https://mail-archives.apache.org/mod_mbox/storm-user/201408.mbox/%3ccajuqm_4kxhsh2_x08ujuqr76m2c+dswp0fcijbmfcaeyqgs...@mail.gmail.com%3E
> Comment from Bobby
> http://mail-archives.apache.org/mod_mbox/storm-user/201501.mbox/%3c574363643.2791948.1420470097280.javamail.ya...@jws10027.mail.ne1.yahoo.com%3E
> {quote}
> What version of storm are you using? Are any of the bolts shell bolts?
> There is a known
> issue where this can happen if two shell bolts share an executor, because
> they are multi-threaded.
> - Bobby
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-770) NullPointerException in consumeBatchToCursor
[
https://issues.apache.org/jira/browse/STORM-770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14508987#comment-14508987
]
Jungtaek Lim commented on STORM-770:
Great!
I'm also looking forward to see it. Thanks for follow up again! :)
> NullPointerException in consumeBatchToCursor
>
>
> Key: STORM-770
> URL: https://issues.apache.org/jira/browse/STORM-770
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 0.9.2-incubating
>Reporter: Stas Levin
>
> We got the following exception after our topology had been up for ~2 days,
> and I was wondering if it might be related.
> Looks like "task" in "mk-transfer-fn" is null, making "(.add remote
> (TaskMessage. task (.serialize serializer tuple)))" fail on NPE
> (worker.clj:128, storm-core-0.9.2-incubating.jar)
> java.lang.RuntimeException: java.lang.NullPointerException
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$consume_loop_STAR_$fn__758.invoke(disruptor.clj:94)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_72]
> Caused by: java.lang.NullPointerException: null
> at clojure.lang.RT.intCast(RT.java:1087) ~[clojure-1.5.1.jar:na]
> at
> backtype.storm.daemon.worker$mk_transfer_fn$fn__5748.invoke(worker.clj:128)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.executor$start_batch_transfer_GT_worker_handler_BANG$fn__5483.invoke(executor.clj:256)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> ... 6 common frames omitted,java.lang.RuntimeException:
> java.lang.NullPointerException
> Any ideas?
> P.S.
> Also saw it here:
> http://mail-archives.apache.org/mod_mbox/storm-user/201501.mbox/%3CCABcMBhCusXXU=v1e66wfuatgyh1euqnd1siog65-tp8xlwx...@mail.gmail.com%3E
> https://mail-archives.apache.org/mod_mbox/storm-user/201408.mbox/%3ccajuqm_4kxhsh2_x08ujuqr76m2c+dswp0fcijbmfcaeyqgs...@mail.gmail.com%3E
> Comment from Bobby
> http://mail-archives.apache.org/mod_mbox/storm-user/201501.mbox/%3c574363643.2791948.1420470097280.javamail.ya...@jws10027.mail.ne1.yahoo.com%3E
> {quote}
> What version of storm are you using? Are any of the bolts shell bolts?
> There is a known
> issue where this can happen if two shell bolts share an executor, because
> they are multi-threaded.
> - Bobby
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Why is Storm Config not modifiable?
I understand that. In my case, I use a class hierarchy with abstract spouts/bolts and multiple concrete spouts/bolts. The abstract class uses some default values (if not provided in the given config) similar to the derived classes. However, some derived classes overwrite the default values for the abstract class, ie, if no value is given in the config, some derived classes set the values and other do not. Additionally, open()/prepare() of the abstract class is called, too, and set the default value if the value was neither provided by the user, not by the derived class. Does it make sense how I explained it? -Matthias On 04/23/2015 01:08 PM, Nathan Leung wrote: > Config should be set when the topology is created before it's sent to the > nimbus. If you write an option to the map there is no mechanism to > propagate the change to the rest of the instances of storm components. > On Apr 23, 2015 6:02 AM, "Matthias J. Sax" > wrote: > >> Hi, >> >> in the Spout/Bolt open()/prepare() method, a Map containing the current >> configuration is given. This Map is of type >> clojure.lang.PersistentHashMap and calling .put(...) raises an >> UnsupportedOperationException. >> >> I was wondering, why a persistent map is used? I could image, that >> system defined values should not be modified. But adding new key/value >> pairs is not allowed either (and would be very helpful in my case). >> >> Right now, I simply copy all values into a new HashMap object and add my >> own values to this new HashMap. But this is an annoying workaround I >> have to do each time... >> >> Would it make sense, the change the type of the system provided Map (or >> modify it's behavior to all for adding new key/value pairs)? >> >> >> -Matthias >> >> >> >> >> > signature.asc Description: OpenPGP digital signature
Re: Why is Storm Config not modifiable?
Config should be set when the topology is created before it's sent to the nimbus. If you write an option to the map there is no mechanism to propagate the change to the rest of the instances of storm components. On Apr 23, 2015 6:02 AM, "Matthias J. Sax" wrote: > Hi, > > in the Spout/Bolt open()/prepare() method, a Map containing the current > configuration is given. This Map is of type > clojure.lang.PersistentHashMap and calling .put(...) raises an > UnsupportedOperationException. > > I was wondering, why a persistent map is used? I could image, that > system defined values should not be modified. But adding new key/value > pairs is not allowed either (and would be very helpful in my case). > > Right now, I simply copy all values into a new HashMap object and add my > own values to this new HashMap. But this is an annoying workaround I > have to do each time... > > Would it make sense, the change the type of the system provided Map (or > modify it's behavior to all for adding new key/value pairs)? > > > -Matthias > > > > >
Why is Storm Config not modifiable?
Hi, in the Spout/Bolt open()/prepare() method, a Map containing the current configuration is given. This Map is of type clojure.lang.PersistentHashMap and calling .put(...) raises an UnsupportedOperationException. I was wondering, why a persistent map is used? I could image, that system defined values should not be modified. But adding new key/value pairs is not allowed either (and would be very helpful in my case). Right now, I simply copy all values into a new HashMap object and add my own values to this new HashMap. But this is an annoying workaround I have to do each time... Would it make sense, the change the type of the system provided Map (or modify it's behavior to all for adding new key/value pairs)? -Matthias signature.asc Description: OpenPGP digital signature
[jira] [Comment Edited] (STORM-770) NullPointerException in consumeBatchToCursor
[
https://issues.apache.org/jira/browse/STORM-770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14508718#comment-14508718
]
Michael Pershyn edited comment on STORM-770 at 4/23/15 9:14 AM:
Thanks :)
I have extended the patch a bit and applied.
There is a {{backtype.storm.daemon.worker/mk-transfer-local-fn}}, and I
observed, that it produces quite frequently the "Received invalid messages for
unknown tasks. Dropping...".
I am really curious, so I extended the {{mk-transfer-local-fn}} to print a bit
more info about the "unknown" tuple-batch.
{code}
;...
(log-warn "Received invalid messages for unknown tasks. Dropping...
tuple-batch type: " (type tuple-batch) ", value: " tuple-batch)
;...
{code}
So, now the storm-0.9.3 with this patch is running. Logs are monitored, so I am
looking forward to get more information.
Thanks for your work ;-)
was (Author: pershyn):
Thanks :)
I have extended the patch a bit and applied.
There is a {{backtype.storm.daemon.worker/mk-transfer-local-fn}}, and I
observed, that produces quite frequently the "Received invalid messages for
unknown tasks. Dropping...".
I am really curious, so I extended the {{mk-transfer-local-fn}} to print a bit
more info about the "unknown" tuple-batch.
{code}
;...
(log-warn "Received invalid messages for unknown tasks. Dropping...
tuple-batch type: " (type tuple-batch) ", value: " tuple-batch)
;...
{code}
So, now the storm-0.9.3 with this patch is running. Logs are monitored, so I am
looking forward to get more information.
Thanks for your work ;-)
> NullPointerException in consumeBatchToCursor
>
>
> Key: STORM-770
> URL: https://issues.apache.org/jira/browse/STORM-770
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 0.9.2-incubating
>Reporter: Stas Levin
>
> We got the following exception after our topology had been up for ~2 days,
> and I was wondering if it might be related.
> Looks like "task" in "mk-transfer-fn" is null, making "(.add remote
> (TaskMessage. task (.serialize serializer tuple)))" fail on NPE
> (worker.clj:128, storm-core-0.9.2-incubating.jar)
> java.lang.RuntimeException: java.lang.NullPointerException
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$consume_loop_STAR_$fn__758.invoke(disruptor.clj:94)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_72]
> Caused by: java.lang.NullPointerException: null
> at clojure.lang.RT.intCast(RT.java:1087) ~[clojure-1.5.1.jar:na]
> at
> backtype.storm.daemon.worker$mk_transfer_fn$fn__5748.invoke(worker.clj:128)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.executor$start_batch_transfer_GT_worker_handler_BANG$fn__5483.invoke(executor.clj:256)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> ... 6 common frames omitted,java.lang.RuntimeException:
> java.lang.NullPointerException
> Any ideas?
> P.S.
> Also saw it here:
> http://mail-archives.apache.org/mod_mbox/storm-user/201501.mbox/%3CCABcMBhCusXXU=v1e66wfuatgyh1euqnd1siog65-tp8xlwx...@mail.gmail.com%3E
> https://mail-archives.apache.org/mod_mbox/storm-user/201408.mbox/%3ccajuqm_4kxhsh2_x08ujuqr76m2c+dswp0fcijbmfcaeyqgs...@mail.gmail.com%3E
> Comment from Bobby
> http://mail-archives.apache.org/mod_mbox/storm-user/201501.mbox/%3c574363643.2791948.1420470097280.javamail.ya...@jws10027.mail.ne1.yahoo.com%3E
> {quote}
> What version of storm are you using? Are any of the bolts shell bolts?
> There is a known
> issue where this can happen if two shell bolts share an executor, because
> they are multi-threaded.
> - Bobby
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-770) NullPointerException in consumeBatchToCursor
[
https://issues.apache.org/jira/browse/STORM-770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14508718#comment-14508718
]
Michael Pershyn commented on STORM-770:
---
Thanks :)
I have extended the patch a bit and applied.
There is a {{backtype.storm.daemon.worker/mk-transfer-local-fn}}, and I
observed, that produces quite frequently the "Received invalid messages for
unknown tasks. Dropping...".
I am really curious, so I extended the {{mk-transfer-local-fn}} to print a bit
more info about the "unknown" tuple-batch.
{code}
;...
(log-warn "Received invalid messages for unknown tasks. Dropping...
tuple-batch type: " (type tuple-batch) ", value: " tuple-batch)
;...
{code}
So, now the storm-0.9.3 with this patch is running. Logs are monitored, so I am
looking forward to get more information.
Thanks for your work ;-)
> NullPointerException in consumeBatchToCursor
>
>
> Key: STORM-770
> URL: https://issues.apache.org/jira/browse/STORM-770
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 0.9.2-incubating
>Reporter: Stas Levin
>
> We got the following exception after our topology had been up for ~2 days,
> and I was wondering if it might be related.
> Looks like "task" in "mk-transfer-fn" is null, making "(.add remote
> (TaskMessage. task (.serialize serializer tuple)))" fail on NPE
> (worker.clj:128, storm-core-0.9.2-incubating.jar)
> java.lang.RuntimeException: java.lang.NullPointerException
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$consume_loop_STAR_$fn__758.invoke(disruptor.clj:94)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_72]
> Caused by: java.lang.NullPointerException: null
> at clojure.lang.RT.intCast(RT.java:1087) ~[clojure-1.5.1.jar:na]
> at
> backtype.storm.daemon.worker$mk_transfer_fn$fn__5748.invoke(worker.clj:128)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.executor$start_batch_transfer_GT_worker_handler_BANG$fn__5483.invoke(executor.clj:256)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> ... 6 common frames omitted,java.lang.RuntimeException:
> java.lang.NullPointerException
> Any ideas?
> P.S.
> Also saw it here:
> http://mail-archives.apache.org/mod_mbox/storm-user/201501.mbox/%3CCABcMBhCusXXU=v1e66wfuatgyh1euqnd1siog65-tp8xlwx...@mail.gmail.com%3E
> https://mail-archives.apache.org/mod_mbox/storm-user/201408.mbox/%3ccajuqm_4kxhsh2_x08ujuqr76m2c+dswp0fcijbmfcaeyqgs...@mail.gmail.com%3E
> Comment from Bobby
> http://mail-archives.apache.org/mod_mbox/storm-user/201501.mbox/%3c574363643.2791948.1420470097280.javamail.ya...@jws10027.mail.ne1.yahoo.com%3E
> {quote}
> What version of storm are you using? Are any of the bolts shell bolts?
> There is a known
> issue where this can happen if two shell bolts share an executor, because
> they are multi-threaded.
> - Bobby
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (STORM-713) Storm Kafka Metrics Don't Include Topic
[
https://issues.apache.org/jira/browse/STORM-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Noll updated STORM-713:
---
Fix Version/s: 0.11.0
> Storm Kafka Metrics Don't Include Topic
> ---
>
> Key: STORM-713
> URL: https://issues.apache.org/jira/browse/STORM-713
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka
>Affects Versions: 0.9.3, 0.10.0
>Reporter: Craig Hawco
>Assignee: Craig Hawco
>Priority: Minor
> Fix For: 0.11.0
>
>
> Storm Kafka does a good job of tracking internal metrics, such as spout
> backlog ("spoutLag"), and contains both aggregate and per-partition values.
> Unfortunately, this data is not tied to a specific topic, and provides no way
> to disambiguate between multiple topics when more than one spout is present.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (STORM-713) Storm Kafka Metrics Don't Include Topic
[
https://issues.apache.org/jira/browse/STORM-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Noll closed STORM-713.
--
Resolution: Fixed
> Storm Kafka Metrics Don't Include Topic
> ---
>
> Key: STORM-713
> URL: https://issues.apache.org/jira/browse/STORM-713
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka
>Affects Versions: 0.9.3, 0.10.0
>Reporter: Craig Hawco
>Assignee: Craig Hawco
>Priority: Minor
> Fix For: 0.11.0
>
>
> Storm Kafka does a good job of tracking internal metrics, such as spout
> backlog ("spoutLag"), and contains both aggregate and per-partition values.
> Unfortunately, this data is not tied to a specific topic, and provides no way
> to disambiguate between multiple topics when more than one spout is present.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
