[jira] [Updated] (STORM-2666) Storm-kafka-client spout can sometimes emit messages that were already committed.
[
https://issues.apache.org/jira/browse/STORM-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated STORM-2666:
--
Labels: pull-request-available (was: )
> Storm-kafka-client spout can sometimes emit messages that were already
> committed.
> --
>
> Key: STORM-2666
> URL: https://issues.apache.org/jira/browse/STORM-2666
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-client
>Affects Versions: 1.0.0, 2.0.0, 1.1.0, 1.1.1, 1.2.0
>Reporter: Guang Du
>Assignee: Stig Rohde Døssing
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Under a certain heavy load, for failed/timeout tuples, the retry service will
> ack tuple for failed max times. Kafka Client Spout will commit after reached
> the commit interval. However seems some 'on the way' tuples will be failed
> again, the retry service will cause Spout to emit again, and acked eventually
> to OffsetManager.
> In some cases such offsets are too many, exceeding the max-uncommit, causing
> org.apache.storm.kafka.spout.internal.OffsetManager#findNextCommitOffset
> unable to find next commit point, and Spout for this partition will not poll
> any more.
> By the way I've applied STORM-2549 PR#2156 from Stig Døssing to fix
> STORM-2625, and I'm using Python Shell Bolt as processing bolt, if this
> information helps.
> resulting logs like below. I'm not sure if the issue has already been
> raised/fixed, glad if anyone could help to point out existing JIRA. Thank you.
> 2017-07-27 22:23:48.398 o.a.s.k.s.KafkaSpout Thread-23-spout-executor[248
> 248] [INFO] Successful ack for tuple message
> [{topic-partition=kafka_bd_trigger_action-20, offset=18204, numFails=0}].
> 2017-07-27 22:23:49.203 o.a.s.k.s.i.OffsetManager
> Thread-23-spout-executor[248 248] [WARN] topic-partition
> [kafka_bd_trigger_action-18] has unexpected offset [16002]. Current committed
> Offset [16003]
> Edit:
> See
> https://issues.apache.org/jira/browse/STORM-2666?focusedCommentId=16125893=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16125893
> for the current best guess at the root cause of this issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (STORM-2357) add At-Most-Once guarantee in KafkaSpout
[ https://issues.apache.org/jira/browse/STORM-2357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stig Rohde Døssing resolved STORM-2357. --- Resolution: Fixed Fix Version/s: 1.2.0 > add At-Most-Once guarantee in KafkaSpout > > > Key: STORM-2357 > URL: https://issues.apache.org/jira/browse/STORM-2357 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka-client >Reporter: Xu Mingmin >Assignee: Stig Rohde Døssing > Fix For: 2.0.0, 1.2.0 > > > KafkaSpout cannot guarantee exactly at-most-once semantic, as it commits > offset periodically, and KafkaSpout.close() is not guarantee to call when a > crash happens. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2648) Kafka spout can't show acks/fails and complete latency when auto commit is enabled
[ https://issues.apache.org/jira/browse/STORM-2648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stig Rohde Døssing resolved STORM-2648. --- Resolution: Fixed Fix Version/s: 1.2.0 > Kafka spout can't show acks/fails and complete latency when auto commit is > enabled > -- > > Key: STORM-2648 > URL: https://issues.apache.org/jira/browse/STORM-2648 > Project: Apache Storm > Issue Type: New Feature > Components: storm-kafka-client >Affects Versions: 2.0.0, 1.1.0, 1.2.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing >Priority: Minor > Labels: pull-request-available > Fix For: 2.0.0, 1.2.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > > The storm-kafka-client spout currently emits tuples with no message ids if > auto commit is enabled. This causes the ack/fail/complete latency counters in > Storm UI to be 0. In some cases this is desirable because the user may not > care, and doesn't want the overhead of Storm tracking tuples. > [~avermeerbergen] expressed a desire to be able to use auto commit without > these counters being disabled, presumably to monitor topology performance. > We should add a toggle that allows users to enable/disable tuple anchoring in > the auto commit case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2765) Disallow colons in blobstore key name
[ https://issues.apache.org/jira/browse/STORM-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-2765. Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~ethanli], I merged this into master. > Disallow colons in blobstore key name > - > > Key: STORM-2765 > URL: https://issues.apache.org/jira/browse/STORM-2765 > Project: Apache Storm > Issue Type: Bug >Reporter: Ethan Li >Assignee: Ethan Li >Priority: Minor > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > We should disallow the use of colons in blobstore key name because HDFS > doesn't support it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2767) Surefire now truncates too much of the stack trace
[
https://issues.apache.org/jira/browse/STORM-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans resolved STORM-2767.
Resolution: Fixed
Fix Version/s: 2.0.0
Thanks [~Srdo],
I merged this into master.
> Surefire now truncates too much of the stack trace
> --
>
> Key: STORM-2767
> URL: https://issues.apache.org/jira/browse/STORM-2767
> Project: Apache Storm
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
>Priority: Minor
> Labels: pull-request-available
> Fix For: 2.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Surefire is truncating so much of the stack trace when tests fail that we
> often can't easily spot the error. As an example I manually threw an NPE from
> storm-kafka-client's KafkaSpout.commit() method, and here are the stack
> traces with trimStackTrace enabled and disabled:
> trimmed
> {code}
> testCommitSuccessWithOffsetVoids(org.apache.storm.kafka.spout.KafkaSpoutCommitTest)
> Time elapsed: 0.714 sec <<< ERROR!
> java.lang.NullPointerException: This is an NPE from inside nextTuple
> at
> org.apache.storm.kafka.spout.KafkaSpoutCommitTest.testCommitSuccessWithOffsetVoids(KafkaSpoutCommitTest.java:87)
> {code}
> not trimmed
> {code}
> testCommitSuccessWithOffsetVoids(org.apache.storm.kafka.spout.KafkaSpoutCommitTest)
> Time elapsed: 0.78 sec <<< ERROR!
> java.lang.NullPointerException: This is an NPE from inside nextTuple
> at org.apache.storm.kafka.spout.KafkaSpout.commit(KafkaSpout.java:266)
> at
> org.apache.storm.kafka.spout.KafkaSpout.nextTuple(KafkaSpout.java:235)
> at
> org.apache.storm.kafka.spout.KafkaSpoutCommitTest.testCommitSuccessWithOffsetVoids(KafkaSpoutCommitTest.java:87)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.junit.runners.Suite.runChild(Suite.java:127)
> at org.junit.runners.Suite.runChild(Suite.java:26)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:161)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)
>
[jira] [Closed] (STORM-2768) Sorry I broke the build
[
https://issues.apache.org/jira/browse/STORM-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans closed STORM-2768.
--
Resolution: Invalid
Never mind. User error
> Sorry I broke the build
> ---
>
> Key: STORM-2768
> URL: https://issues.apache.org/jira/browse/STORM-2768
> Project: Apache Storm
> Issue Type: Bug
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
>
> Not totally sure what happened, but when I try to build I am getting an error
> like
> {code}
> [ERROR] The project org.apache.storm:blobstore-migrator:2.0.0-SNAPSHOT
> (/home/evans/src/storm/external/storm-blobstore-migration/pom.xml) has 1 error
> [ERROR] 'dependencies.dependency.version' for
> org.apache.hadoop:hadoop-hdfs:jar must be a valid version but is
> '${hdfs.version}'. @ line 64, column 22
> {code}
> I will fix it
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (STORM-2768) Sorry I broke the build
Robert Joseph Evans created STORM-2768:
--
Summary: Sorry I broke the build
Key: STORM-2768
URL: https://issues.apache.org/jira/browse/STORM-2768
Project: Apache Storm
Issue Type: Bug
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
Priority: Blocker
Not totally sure what happened, but when I try to build I am getting an error
like
{code}
[ERROR] The project org.apache.storm:blobstore-migrator:2.0.0-SNAPSHOT
(/home/evans/src/storm/external/storm-blobstore-migration/pom.xml) has 1 error
[ERROR] 'dependencies.dependency.version' for
org.apache.hadoop:hadoop-hdfs:jar must be a valid version but is
'${hdfs.version}'. @ line 64, column 22
{code}
I will fix it
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (STORM-2756) STORM-2548 on 1.x-branch broke setting key/value deserializers with the now deprecated setKey/setValue methods
[ https://issues.apache.org/jira/browse/STORM-2756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2756. - Resolution: Fixed Fix Version/s: 1.2.0 Thanks [~Srdo], I merged into 1.x branch. > STORM-2548 on 1.x-branch broke setting key/value deserializers with the now > deprecated setKey/setValue methods > -- > > Key: STORM-2756 > URL: https://issues.apache.org/jira/browse/STORM-2756 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka-client >Affects Versions: 1.2.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing > Labels: pull-request-available > Fix For: 1.2.0 > > Time Spent: 20m > Remaining Estimate: 0h > > When STORM-2548 was backported, the setKey/setValue methods on > KafkaSpoutConfig.Builder were deprecated, and users were directed to use > setProp along with the relevant ConsumerConfig constants for setting > deserializers instead. > As part of this change, the KafkaConsumerFactoryDefault switched from using > the KafkaConsumer(props, keyDes, valDes) constructor to using the > KafkaConsumer(props) constructor. Unfortunately I forgot to update the > KafkaSpoutConfig.Builder constructor properly, so if the user configures the > deserializer via either the Builder constructor parameters or > setKey/setValue, the setting is not put in the kafkaProps map and the > deserializer is not used. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2718) Show some descriptions on LogViewer index page
[ https://issues.apache.org/jira/browse/STORM-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2718. - Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~ethanli], I merged into master. > Show some descriptions on LogViewer index page > -- > > Key: STORM-2718 > URL: https://issues.apache.org/jira/browse/STORM-2718 > Project: Apache Storm > Issue Type: Improvement >Reporter: Ethan Li >Assignee: Ethan Li >Priority: Minor > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > I remembered when I was pretty new to storm and I launched logviewer for the > first time. I visited x:8000 out of my natural but then got "page not > found". I thought there must be something wrong with my storm setup. It took > me a while to figure it out that logviewer doesn't work that way. I also > found some other people were also confused about this, > [example|https://stackoverflow.com/questions/25538327/storm-logviewer-page-not-found] > . We probably want to show some descriptions/guides on the index page. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2764) HDFSBlobStore leaks file system objects
[ https://issues.apache.org/jira/browse/STORM-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2764. - Resolution: Fixed Fix Version/s: 1.0.6 1.1.2 1.2.0 2.0.0 Thanks [~revans2], I merged into master, 1.x, 1.1.x, 1.0.x branches. > HDFSBlobStore leaks file system objects > --- > > Key: STORM-2764 > URL: https://issues.apache.org/jira/browse/STORM-2764 > Project: Apache Storm > Issue Type: Bug > Components: storm-hdfs >Affects Versions: 1.0.0, 2.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Labels: pull-request-available > Fix For: 2.0.0, 1.2.0, 1.1.2, 1.0.6 > > Time Spent: 1h > Remaining Estimate: 0h > > This impacts all of the releases. Each time we create a new HDFSBlobStore > instance we call > https://github.com/apache/storm/blob/v1.0.0/external/storm-hdfs/src/main/java/org/apache/storm/hdfs/blobstore/HdfsBlobStore.java#L140 > loginUserFromKeytab. > This results in a new subject being created each time when ends up causing a > FileSystem object to leak each time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
