[jira] [Commented] (KAFKA-2236) offset request reply racing with segment rolling
[ https://issues.apache.org/jira/browse/KAFKA-2236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271465#comment-15271465 ] ASF GitHub Bot commented on KAFKA-2236: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1318 > offset request reply racing with segment rolling > > > Key: KAFKA-2236 > URL: https://issues.apache.org/jira/browse/KAFKA-2236 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8.2.0 > Environment: Linux x86_64, java.1.7.0_72, discovered using librdkafka > based client. >Reporter: Alfred Landrum >Assignee: Jason Gustafson >Priority: Critical > Labels: newbie > > My use case with kafka involves an aggressive retention policy that rolls > segment files frequently. My librdkafka based client sees occasional errors > to offset requests, showing up in the broker log like: > [2015-06-02 02:33:38,047] INFO Rolled new log segment for > 'receiver-93b40462-3850-47c1-bcda-8a3e221328ca-50' in 1 ms. (kafka.log.Log) > [2015-06-02 02:33:38,049] WARN [KafkaApi-0] Error while responding to offset > request (kafka.server.KafkaApis) > java.lang.ArrayIndexOutOfBoundsException: 3 > at kafka.server.KafkaApis.fetchOffsetsBefore(KafkaApis.scala:469) > at kafka.server.KafkaApis.fetchOffsets(KafkaApis.scala:449) > at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:411) > at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:402) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at scala.collection.immutable.Map$Map1.foreach(Map.scala:109) > at > scala.collection.TraversableLike$class.map(TraversableLike.scala:244) > at scala.collection.AbstractTraversable.map(Traversable.scala:105) > at kafka.server.KafkaApis.handleOffsetRequest(KafkaApis.scala:402) > at kafka.server.KafkaApis.handle(KafkaApis.scala:61) > at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59) > at java.lang.Thread.run(Thread.java:745) > quoting Guozhang Wang's reply to my query on the users list: > "I check the 0.8.2 code and may probably find a bug related to your issue. > Basically, segsArray.last.size is called multiple times during handling > offset requests, while segsArray.last could get concurrent appends. Hence > it is possible that in line 461, if(segsArray.last.size > 0) returns false > while later in line 468, if(segsArray.last.size > 0) could return true." > http://mail-archives.apache.org/mod_mbox/kafka-users/201506.mbox/%3CCAHwHRrUK-3wdoEAaFbsD0E859Ea0gXixfxgDzF8E3%3D_8r7K%2Bpw%40mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2236) offset request reply racing with segment rolling
[ https://issues.apache.org/jira/browse/KAFKA-2236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15270380#comment-15270380 ] ASF GitHub Bot commented on KAFKA-2236: --- GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/1318 KAFKA-2236; Offset request reply racing with segment rolling You can merge this pull request into a Git repository by running: $ git pull https://github.com/ijuma/kafka KAFKA-2236-offset-request-reply-segment-rolling-race Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1318.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1318 commit 7d6d50cd377de82215856e22fc09e137a724acfc Author: William Thurston Date: 2015-07-19T02:54:05Z Addresses Jira Kafka-2236 to eliminate an array index out of bounds exception when segments are appended between checks > offset request reply racing with segment rolling > > > Key: KAFKA-2236 > URL: https://issues.apache.org/jira/browse/KAFKA-2236 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8.2.0 > Environment: Linux x86_64, java.1.7.0_72, discovered using librdkafka > based client. >Reporter: Alfred Landrum >Assignee: Jason Gustafson >Priority: Critical > Labels: newbie > > My use case with kafka involves an aggressive retention policy that rolls > segment files frequently. My librdkafka based client sees occasional errors > to offset requests, showing up in the broker log like: > [2015-06-02 02:33:38,047] INFO Rolled new log segment for > 'receiver-93b40462-3850-47c1-bcda-8a3e221328ca-50' in 1 ms. (kafka.log.Log) > [2015-06-02 02:33:38,049] WARN [KafkaApi-0] Error while responding to offset > request (kafka.server.KafkaApis) > java.lang.ArrayIndexOutOfBoundsException: 3 > at kafka.server.KafkaApis.fetchOffsetsBefore(KafkaApis.scala:469) > at kafka.server.KafkaApis.fetchOffsets(KafkaApis.scala:449) > at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:411) > at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:402) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at scala.collection.immutable.Map$Map1.foreach(Map.scala:109) > at > scala.collection.TraversableLike$class.map(TraversableLike.scala:244) > at scala.collection.AbstractTraversable.map(Traversable.scala:105) > at kafka.server.KafkaApis.handleOffsetRequest(KafkaApis.scala:402) > at kafka.server.KafkaApis.handle(KafkaApis.scala:61) > at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59) > at java.lang.Thread.run(Thread.java:745) > quoting Guozhang Wang's reply to my query on the users list: > "I check the 0.8.2 code and may probably find a bug related to your issue. > Basically, segsArray.last.size is called multiple times during handling > offset requests, while segsArray.last could get concurrent appends. Hence > it is possible that in line 461, if(segsArray.last.size > 0) returns false > while later in line 468, if(segsArray.last.size > 0) could return true." > http://mail-archives.apache.org/mod_mbox/kafka-users/201506.mbox/%3CCAHwHRrUK-3wdoEAaFbsD0E859Ea0gXixfxgDzF8E3%3D_8r7K%2Bpw%40mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2236) offset request reply racing with segment rolling
[ https://issues.apache.org/jira/browse/KAFKA-2236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15149808#comment-15149808 ] ASF GitHub Bot commented on KAFKA-2236: --- Github user jhspaybar closed the pull request at: https://github.com/apache/kafka/pull/86 > offset request reply racing with segment rolling > > > Key: KAFKA-2236 > URL: https://issues.apache.org/jira/browse/KAFKA-2236 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8.2.0 > Environment: Linux x86_64, java.1.7.0_72, discovered using librdkafka > based client. >Reporter: Alfred Landrum >Assignee: Jason Gustafson >Priority: Critical > Labels: newbie > > My use case with kafka involves an aggressive retention policy that rolls > segment files frequently. My librdkafka based client sees occasional errors > to offset requests, showing up in the broker log like: > [2015-06-02 02:33:38,047] INFO Rolled new log segment for > 'receiver-93b40462-3850-47c1-bcda-8a3e221328ca-50' in 1 ms. (kafka.log.Log) > [2015-06-02 02:33:38,049] WARN [KafkaApi-0] Error while responding to offset > request (kafka.server.KafkaApis) > java.lang.ArrayIndexOutOfBoundsException: 3 > at kafka.server.KafkaApis.fetchOffsetsBefore(KafkaApis.scala:469) > at kafka.server.KafkaApis.fetchOffsets(KafkaApis.scala:449) > at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:411) > at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:402) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at scala.collection.immutable.Map$Map1.foreach(Map.scala:109) > at > scala.collection.TraversableLike$class.map(TraversableLike.scala:244) > at scala.collection.AbstractTraversable.map(Traversable.scala:105) > at kafka.server.KafkaApis.handleOffsetRequest(KafkaApis.scala:402) > at kafka.server.KafkaApis.handle(KafkaApis.scala:61) > at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59) > at java.lang.Thread.run(Thread.java:745) > quoting Guozhang Wang's reply to my query on the users list: > "I check the 0.8.2 code and may probably find a bug related to your issue. > Basically, segsArray.last.size is called multiple times during handling > offset requests, while segsArray.last could get concurrent appends. Hence > it is possible that in line 461, if(segsArray.last.size > 0) returns false > while later in line 468, if(segsArray.last.size > 0) could return true." > http://mail-archives.apache.org/mod_mbox/kafka-users/201506.mbox/%3CCAHwHRrUK-3wdoEAaFbsD0E859Ea0gXixfxgDzF8E3%3D_8r7K%2Bpw%40mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2236) offset request reply racing with segment rolling
[
https://issues.apache.org/jira/browse/KAFKA-2236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14902803#comment-14902803
]
Robert Metzger commented on KAFKA-2236:
---
I've seen this error as well in the integration tests of Flink (We are starting
a Kafka Broker with the test to test our Kafka connectors):
{code}
14:43:03,328 INFO kafka.network.Processor
- Closing socket connection to /127.0.0.1.
14:43:03,334 WARN kafka.server.KafkaApis
- [KafkaApi-0] Error while responding to offset request
java.lang.ArrayIndexOutOfBoundsException: 1
at kafka.server.KafkaApis.fetchOffsetsBefore(KafkaApis.scala:469)
at kafka.server.KafkaApis.fetchOffsets(KafkaApis.scala:449)
at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:411)
at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:402)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at kafka.server.KafkaApis.handleOffsetRequest(KafkaApis.scala:402)
at kafka.server.KafkaApis.handle(KafkaApis.scala:61)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59)
at java.lang.Thread.run(Thread.java:745)
{code}
on the client side, we are seeing a {{kafka.common.UnknownException}}:
{code}
Caused by: java.lang.RuntimeException: Unable to get last offset for topic
customPartitioningTestTopic and partitions [FetchPartition {partition=2,
offset=-915623761776}].
Exception for partition 2: kafka.common.UnknownException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
at java.lang.Class.newInstance(Class.java:438)
at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:86)
at kafka.common.ErrorMapping.exceptionFor(ErrorMapping.scala)
at
org.apache.flink.streaming.connectors.kafka.internals.LegacyFetcher$SimpleConsumerThread.getLastOffset(LegacyFetcher.java:521)
at
org.apache.flink.streaming.connectors.kafka.internals.LegacyFetcher$SimpleConsumerThread.run(LegacyFetcher.java:370)
{code}
Here is the log of the travis build:
https://s3.amazonaws.com/archive.travis-ci.org/jobs/8158/log.txt
this is the archive containing the full logs at INFO level:
https://flink-logs-us.s3.amazonaws.com/travis-artifacts/rmetzger/flink/1123/1123.4.tar.gz
(see 2.log)
It would be nice if you could address the issue with the next release.
> offset request reply racing with segment rolling
>
>
> Key: KAFKA-2236
> URL: https://issues.apache.org/jira/browse/KAFKA-2236
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.8.2.0
> Environment: Linux x86_64, java.1.7.0_72, discovered using librdkafka
> based client.
>Reporter: Alfred Landrum
>Assignee: Jason Gustafson
>Priority: Critical
> Labels: newbie
>
> My use case with kafka involves an aggressive retention policy that rolls
> segment files frequently. My librdkafka based client sees occasional errors
> to offset requests, showing up in the broker log like:
> [2015-06-02 02:33:38,047] INFO Rolled new log segment for
> 'receiver-93b40462-3850-47c1-bcda-8a3e221328ca-50' in 1 ms. (kafka.log.Log)
> [2015-06-02 02:33:38,049] WARN [KafkaApi-0] Error while responding to offset
> request (kafka.server.KafkaApis)
> java.lang.ArrayIndexOutOfBoundsException: 3
> at kafka.server.KafkaApis.fetchOffsetsBefore(KafkaApis.scala:469)
> at kafka.server.KafkaApis.fetchOffsets(KafkaApis.scala:449)
> at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:411)
> at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:402)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
> at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> at scala.collection.AbstractTraversable.map(Traversable.scala:105)
> at
[jira] [Commented] (KAFKA-2236) offset request reply racing with segment rolling
[ https://issues.apache.org/jira/browse/KAFKA-2236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14633748#comment-14633748 ] William Thurston commented on KAFKA-2236: - https://github.com/apache/kafka/pull/86 > offset request reply racing with segment rolling > > > Key: KAFKA-2236 > URL: https://issues.apache.org/jira/browse/KAFKA-2236 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8.2.0 > Environment: Linux x86_64, java.1.7.0_72, discovered using librdkafka > based client. >Reporter: Alfred Landrum >Assignee: Jason Gustafson >Priority: Critical > Labels: newbie > > My use case with kafka involves an aggressive retention policy that rolls > segment files frequently. My librdkafka based client sees occasional errors > to offset requests, showing up in the broker log like: > [2015-06-02 02:33:38,047] INFO Rolled new log segment for > 'receiver-93b40462-3850-47c1-bcda-8a3e221328ca-50' in 1 ms. (kafka.log.Log) > [2015-06-02 02:33:38,049] WARN [KafkaApi-0] Error while responding to offset > request (kafka.server.KafkaApis) > java.lang.ArrayIndexOutOfBoundsException: 3 > at kafka.server.KafkaApis.fetchOffsetsBefore(KafkaApis.scala:469) > at kafka.server.KafkaApis.fetchOffsets(KafkaApis.scala:449) > at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:411) > at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:402) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at scala.collection.immutable.Map$Map1.foreach(Map.scala:109) > at > scala.collection.TraversableLike$class.map(TraversableLike.scala:244) > at scala.collection.AbstractTraversable.map(Traversable.scala:105) > at kafka.server.KafkaApis.handleOffsetRequest(KafkaApis.scala:402) > at kafka.server.KafkaApis.handle(KafkaApis.scala:61) > at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59) > at java.lang.Thread.run(Thread.java:745) > quoting Guozhang Wang's reply to my query on the users list: > "I check the 0.8.2 code and may probably find a bug related to your issue. > Basically, segsArray.last.size is called multiple times during handling > offset requests, while segsArray.last could get concurrent appends. Hence > it is possible that in line 461, if(segsArray.last.size > 0) returns false > while later in line 468, if(segsArray.last.size > 0) could return true." > http://mail-archives.apache.org/mod_mbox/kafka-users/201506.mbox/%3CCAHwHRrUK-3wdoEAaFbsD0E859Ea0gXixfxgDzF8E3%3D_8r7K%2Bpw%40mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
