[jira] [Commented] (KAFKA-12845) Rollback change which requires join key to be non null on KStream->GlobalKTable
[ https://issues.apache.org/jira/browse/KAFKA-12845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17352903#comment-17352903 ] Pedro Gontijo commented on KAFKA-12845: --- Got it. But if you look at the code before the change you will see that its behavior was exactly what you have suggested in (https://issues.apache.org/jira/browse/KAFKA-12317). It was already there. KStream->KTable *and* KStream->GlobalKTable were both ok with null join-keys on left join scenarios. The change proposed by KAFKA-10277 is totally ok, but actually, the code change was beyond the proposed when instead of just removing "key == null" from condition "if (*key* == null || value == null)" at line 69, it made it "if (*mappedKey* == null || value == null)". So, yes, this can be closed, but I would definitely advocate for K12317 to be a bug fix because what we have now in 2.7.x is not only changing behavior (in a negative way I think) but also affecting performance by querying the state store unnecessarily. > Rollback change which requires join key to be non null on > KStream->GlobalKTable > --- > > Key: KAFKA-12845 > URL: https://issues.apache.org/jira/browse/KAFKA-12845 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.7.0 >Reporter: Pedro Gontijo >Priority: Major > > As part of [KAFKA-10277|https://issues.apache.org/jira/browse/KAFKA-10277] > the behavior for KStream->GlobalKtable joins was changed to require non null > join keys. > But it seems reasonable that not every record will have an existing > relationship (and hence a key) with the join globalktable. Think about a > User>Car for instance, or PageView>Product. An empty/zero key could be > returned by the KeyMapper but that will make a totally unnecessary search > into the store. > I do not think that makes sense for any GlobalKtable join (inner or left) but > for left join it sounds even more strange. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-12845) Rollback change which requires join key to be non null on KStream->GlobalKTable
[ https://issues.apache.org/jira/browse/KAFKA-12845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17352111#comment-17352111 ] Pedro Gontijo edited comment on KAFKA-12845 at 5/26/21, 10:43 PM: -- Thanks for the reply [~mjsax] I see what you are saying but that is not the old behavior. Before K10277 the join-key could be null without dropping the record and just pushing it forward. See lines 69 and 70 from this diff [https://github.com/apache/kafka/pull/9186/files#diff-c488ac95d9919270502bd7b52f66147612353d7e8e957233093d3c777fb044a7L69-R70] Also line 77, which is where the mapperKey was actually checked for null. was (Author: pedrong): Thanks for the reply [~mjsax] I see what you are saying but that is not the old behavior. Before K10277 the join-key could be null without dropping the record and just pushing it forward. See lines 69 and 70 from this diff [https://github.com/apache/kafka/pull/9186/files#diff-c488ac95d9919270502bd7b52f66147612353d7e8e957233093d3c777fb044a7L69-R70] > Rollback change which requires join key to be non null on > KStream->GlobalKTable > --- > > Key: KAFKA-12845 > URL: https://issues.apache.org/jira/browse/KAFKA-12845 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.7.0 >Reporter: Pedro Gontijo >Priority: Major > > As part of [KAFKA-10277|https://issues.apache.org/jira/browse/KAFKA-10277] > the behavior for KStream->GlobalKtable joins was changed to require non null > join keys. > But it seems reasonable that not every record will have an existing > relationship (and hence a key) with the join globalktable. Think about a > User>Car for instance, or PageView>Product. An empty/zero key could be > returned by the KeyMapper but that will make a totally unnecessary search > into the store. > I do not think that makes sense for any GlobalKtable join (inner or left) but > for left join it sounds even more strange. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12845) Rollback change which requires join key to be non null on KStream->GlobalKTable
[ https://issues.apache.org/jira/browse/KAFKA-12845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17352111#comment-17352111 ] Pedro Gontijo commented on KAFKA-12845: --- Thanks for the reply [~mjsax] I see what you are saying but that is not the old behavior. Before K10277 the join-key could be null without dropping the record and just pushing it forward. See lines 69 and 70 from this diff [https://github.com/apache/kafka/pull/9186/files#diff-c488ac95d9919270502bd7b52f66147612353d7e8e957233093d3c777fb044a7L69-R70] > Rollback change which requires join key to be non null on > KStream->GlobalKTable > --- > > Key: KAFKA-12845 > URL: https://issues.apache.org/jira/browse/KAFKA-12845 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.7.0 >Reporter: Pedro Gontijo >Priority: Major > > As part of [KAFKA-10277|https://issues.apache.org/jira/browse/KAFKA-10277] > the behavior for KStream->GlobalKtable joins was changed to require non null > join keys. > But it seems reasonable that not every record will have an existing > relationship (and hence a key) with the join globalktable. Think about a > User>Car for instance, or PageView>Product. An empty/zero key could be > returned by the KeyMapper but that will make a totally unnecessary search > into the store. > I do not think that makes sense for any GlobalKtable join (inner or left) but > for left join it sounds even more strange. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-12845) Rollback change which requires join key to be non null on KStream->GlobalKTable
[ https://issues.apache.org/jira/browse/KAFKA-12845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17351227#comment-17351227 ] Pedro Gontijo edited comment on KAFKA-12845 at 5/25/21, 5:26 PM: - [~mjsax] and [~JoelWee] , I would love to hear your thoughts on this. was (Author: pedrong): [~mjsax] and [~JoelWee] would love to hear your thoughts on this. > Rollback change which requires join key to be non null on > KStream->GlobalKTable > --- > > Key: KAFKA-12845 > URL: https://issues.apache.org/jira/browse/KAFKA-12845 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.7.0 >Reporter: Pedro Gontijo >Priority: Major > > As part of [KAFKA-10277|https://issues.apache.org/jira/browse/KAFKA-10277] > the behavior for KStream->GlobalKtable joins was changed to require non null > join keys. > But it seems reasonable that not every record will have an existing > relationship (and hence a key) with the join globalktable. Think about a > User>Car for instance, or PageView>Product. An empty/zero key could be > returned by the KeyMapper but that will make a totally unnecessary search > into the store. > I do not think that makes sense for any GlobalKtable join (inner or left) but > for left join it sounds even more strange. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-12845) Rollback change which requires join key to be non null on KStream->GlobalKTable
[ https://issues.apache.org/jira/browse/KAFKA-12845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17351227#comment-17351227 ] Pedro Gontijo edited comment on KAFKA-12845 at 5/25/21, 5:26 PM: - [~mjsax] and [~JoelWee] would love to hear your thoughts on this. was (Author: pedrong): [~mjsax] and [~JoelWee] it would be great o hear your thoughts on this. > Rollback change which requires join key to be non null on > KStream->GlobalKTable > --- > > Key: KAFKA-12845 > URL: https://issues.apache.org/jira/browse/KAFKA-12845 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.7.0 >Reporter: Pedro Gontijo >Priority: Major > > As part of [KAFKA-10277|https://issues.apache.org/jira/browse/KAFKA-10277] > the behavior for KStream->GlobalKtable joins was changed to require non null > join keys. > But it seems reasonable that not every record will have an existing > relationship (and hence a key) with the join globalktable. Think about a > User>Car for instance, or PageView>Product. An empty/zero key could be > returned by the KeyMapper but that will make a totally unnecessary search > into the store. > I do not think that makes sense for any GlobalKtable join (inner or left) but > for left join it sounds even more strange. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12845) Rollback change which requires join key to be non null on KStream->GlobalKTable
[ https://issues.apache.org/jira/browse/KAFKA-12845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17351227#comment-17351227 ] Pedro Gontijo commented on KAFKA-12845: --- [~mjsax] and [~JoelWee] it would be great o hear your thoughts on this. > Rollback change which requires join key to be non null on > KStream->GlobalKTable > --- > > Key: KAFKA-12845 > URL: https://issues.apache.org/jira/browse/KAFKA-12845 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.7.0 >Reporter: Pedro Gontijo >Priority: Major > > As part of [KAFKA-10277|https://issues.apache.org/jira/browse/KAFKA-10277] > the behavior for KStream->GlobalKtable joins was changed to require non null > join keys. > But it seems reasonable that not every record will have an existing > relationship (and hence a key) with the join globalktable. Think about a > User>Car for instance, or PageView>Product. An empty/zero key could be > returned by the KeyMapper but that will make a totally unnecessary search > into the store. > I do not think that makes sense for any GlobalKtable join (inner or left) but > for left join it sounds even more strange. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-12845) Rollback change which requires join key to be non null on KStream->GlobalKTable
Pedro Gontijo created KAFKA-12845: - Summary: Rollback change which requires join key to be non null on KStream->GlobalKTable Key: KAFKA-12845 URL: https://issues.apache.org/jira/browse/KAFKA-12845 Project: Kafka Issue Type: Improvement Components: streams Affects Versions: 2.7.0 Reporter: Pedro Gontijo As part of [KAFKA-10277|https://issues.apache.org/jira/browse/KAFKA-10277] the behavior for KStream->GlobalKtable joins was changed to require non null join keys. But it seems reasonable that not every record will have an existing relationship (and hence a key) with the join globalktable. Think about a User>Car for instance, or PageView>Product. An empty/zero key could be returned by the KeyMapper but that will make a totally unnecessary search into the store. I do not think that makes sense for any GlobalKtable join (inner or left) but for left join it sounds even more strange. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10493) Ktable out-of-order updates are not being ignored
[
https://issues.apache.org/jira/browse/KAFKA-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17198650#comment-17198650
]
Pedro Gontijo commented on KAFKA-10493:
---
I see. My assumption was basically based on the javadoc at
[https://github.com/apache/kafka/blob/2.6.0/streams/src/main/java/org/apache/kafka/streams/state/internals/ValueAndTimestampSerializer.java#L58],
that says that if the right (new record) timestamp is less than the left
(existing record) then it would return true. Which would make
{{store.putIfDifferentValues}} return false, and hence not update the store and
not forward the record.
On my scenario not having the record updated would be perfect btw. But I do
understand the concern on backward compatibility.
Maybe a configuration parameter in Materialized could be this opt-in. If you
think it makes sense I can try to create a PR.
> Ktable out-of-order updates are not being ignored
> -
>
> Key: KAFKA-10493
> URL: https://issues.apache.org/jira/browse/KAFKA-10493
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.6.0
>Reporter: Pedro Gontijo
>Priority: Major
> Attachments: KTableOutOfOrderBug.java
>
>
> On a materialized KTable, out-of-order records for a given key (records which
> timestamp are older than the current value in store) are not being ignored
> but used to update the local store value and also being forwarded.
> I believe the bug is here:
> [https://github.com/apache/kafka/blob/2.6.0/streams/src/main/java/org/apache/kafka/streams/state/internals/ValueAndTimestampSerializer.java#L77]
> It should return true, not false (see javadoc)
> The bug impacts here:
> [https://github.com/apache/kafka/blob/2.6.0/streams/src/main/java/org/apache/kafka/streams/kstream/internals/KTableSource.java#L142-L148]
> I have attached a simple stream app that shows the issue happening.
> Thank you!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-10493) Ktable out-of-order updates are not being ignored
[ https://issues.apache.org/jira/browse/KAFKA-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pedro Gontijo updated KAFKA-10493: -- Affects Version/s: (was: 2.3.0) > Ktable out-of-order updates are not being ignored > - > > Key: KAFKA-10493 > URL: https://issues.apache.org/jira/browse/KAFKA-10493 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.6.0 >Reporter: Pedro Gontijo >Priority: Major > Attachments: KTableOutOfOrderBug.java > > > On a materialized KTable, out-of-order records for a given key (records which > timestamp are older than the current value in store) are not being ignored > but used to update the local store value and also being forwarded. > I believe the bug is here: > [https://github.com/apache/kafka/blob/2.6.0/streams/src/main/java/org/apache/kafka/streams/state/internals/ValueAndTimestampSerializer.java#L77] > It should return true, not false (see javadoc) > The bug impacts here: > [https://github.com/apache/kafka/blob/2.6.0/streams/src/main/java/org/apache/kafka/streams/kstream/internals/KTableSource.java#L142-L148] > I have attached a simple stream app that shows the issue happening. > Thank you! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-10493) Ktable out-of-order updates are not being ignored
Pedro Gontijo created KAFKA-10493: - Summary: Ktable out-of-order updates are not being ignored Key: KAFKA-10493 URL: https://issues.apache.org/jira/browse/KAFKA-10493 Project: Kafka Issue Type: Bug Components: streams Affects Versions: 2.6.0, 2.3.0 Reporter: Pedro Gontijo Attachments: KTableOutOfOrderBug.java On a materialized KTable, out-of-order records for a given key (records which timestamp are older than the current value in store) are not being ignored but used to update the local store value and also being forwarded. I believe the bug is here: [https://github.com/apache/kafka/blob/2.6.0/streams/src/main/java/org/apache/kafka/streams/state/internals/ValueAndTimestampSerializer.java#L77] It should return true, not false (see javadoc) The bug impacts here: [https://github.com/apache/kafka/blob/2.6.0/streams/src/main/java/org/apache/kafka/streams/kstream/internals/KTableSource.java#L142-L148] I have attached a simple stream app that shows the issue happening. Thank you! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-7757) Too many open files after java.io.IOException: Connection to n was disconnected before the response was read
[
https://issues.apache.org/jira/browse/KAFKA-7757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pedro Gontijo updated KAFKA-7757:
-
Description:
We upgraded from 0.10.2.2 to 2.1.0 (a cluster with 3 brokers)
After a while (hours) 2 brokers start to throw:
{code:java}
java.io.IOException: Connection to NN was disconnected before the response was
read
at
org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:97)
at
kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:97)
at
kafka.server.ReplicaFetcherThread.fetchFromLeader(ReplicaFetcherThread.scala:190)
at
kafka.server.AbstractFetcherThread.kafka$server$AbstractFetcherThread$$processFetchRequest(AbstractFetcherThread.scala:241)
at
kafka.server.AbstractFetcherThread$$anonfun$maybeFetch$1.apply(AbstractFetcherThread.scala:130)
at
kafka.server.AbstractFetcherThread$$anonfun$maybeFetch$1.apply(AbstractFetcherThread.scala:129)
at scala.Option.foreach(Option.scala:257)
at
kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:129)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:111)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
{code}
File descriptors start to pile up and if I do not restart it throws "Too many
open files" and crashes.
{code:java}
ERROR Error while accepting connection (kafka.network.Acceptor)
java.io.IOException: Too many open files in system
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at kafka.network.Acceptor.accept(SocketServer.scala:460)
at kafka.network.Acceptor.run(SocketServer.scala:403)
at java.lang.Thread.run(Thread.java:748)
{code}
After some hours the issue happens again... It has happened with all brokers,
so it is not something specific to an instance.
was:
We upgraded from 0.10.2.2 to 2.1.0 (a cluster with 3 brokers)
After a while (hours) 2 brokers start to throw:
{code:java}
java.io.IOException: Connection to NN was disconnected before the response was
read
at
org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:97)
at
kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:97)
at
kafka.server.ReplicaFetcherThread.fetchFromLeader(ReplicaFetcherThread.scala:190)
at
kafka.server.AbstractFetcherThread.kafka$server$AbstractFetcherThread$$processFetchRequest(AbstractFetcherThread.scala:241)
at
kafka.server.AbstractFetcherThread$$anonfun$maybeFetch$1.apply(AbstractFetcherThread.scala:130)
at
kafka.server.AbstractFetcherThread$$anonfun$maybeFetch$1.apply(AbstractFetcherThread.scala:129)
at scala.Option.foreach(Option.scala:257)
at
kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:129)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:111)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
{code}
The problem has happened with all brokers.
File descriptors start to pile up and if I do not restart it throws "Too many
open files" and crashes.
{code:java}
ERROR Error while accepting connection (kafka.network.Acceptor)
java.io.IOException: Too many open files in system
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at kafka.network.Acceptor.accept(SocketServer.scala:460)
at kafka.network.Acceptor.run(SocketServer.scala:403)
at java.lang.Thread.run(Thread.java:748)
{code}
After some hours the issue happens again...
> Too many open files after java.io.IOException: Connection to n was
> disconnected before the response was read
>
>
> Key: KAFKA-7757
> URL: https://issues.apache.org/jira/browse/KAFKA-7757
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 2.1.0
>Reporter: Pedro Gontijo
>Priority: Major
> Attachments: server.properties, td1.txt, td2.txt, td3.txt
>
>
> We upgraded from 0.10.2.2 to 2.1.0 (a cluster with 3 brokers)
> After a while (hours) 2 brokers start to throw:
> {code:java}
> java.io.IOException: Connection to NN was disconnected before the response
> was read
> at
> org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:97)
> at
> kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:97)
> at
> kafka.server.ReplicaFetcherThread.fetchFromLeader(ReplicaFetcherThread.scala:190)
> at
> kafka.server.AbstractFetcherThread.kafka$server$AbstractFetcherTh
[jira] [Updated] (KAFKA-7757) Too many open files after java.io.IOException: Connection to n was disconnected before the response was read
[
https://issues.apache.org/jira/browse/KAFKA-7757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pedro Gontijo updated KAFKA-7757:
-
Attachment: server.properties
> Too many open files after java.io.IOException: Connection to n was
> disconnected before the response was read
>
>
> Key: KAFKA-7757
> URL: https://issues.apache.org/jira/browse/KAFKA-7757
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 2.1.0
>Reporter: Pedro Gontijo
>Priority: Major
> Attachments: server.properties
>
>
> We upgraded from 0.10.2.2 to 2.1.0 (a cluster with 3 brokers)
> After a while (hours) 2 brokers start to throw:
> {code:java}
> java.io.IOException: Connection to NN was disconnected before the response
> was read
> at
> org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:97)
> at
> kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:97)
> at
> kafka.server.ReplicaFetcherThread.fetchFromLeader(ReplicaFetcherThread.scala:190)
> at
> kafka.server.AbstractFetcherThread.kafka$server$AbstractFetcherThread$$processFetchRequest(AbstractFetcherThread.scala:241)
> at
> kafka.server.AbstractFetcherThread$$anonfun$maybeFetch$1.apply(AbstractFetcherThread.scala:130)
> at
> kafka.server.AbstractFetcherThread$$anonfun$maybeFetch$1.apply(AbstractFetcherThread.scala:129)
> at scala.Option.foreach(Option.scala:257)
> at
> kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:129)
> at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:111)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
> {code}
> The problem has happened with all brokers.
> File descriptors start to pile up and if I do not restart it throws "Too many
> open files" and crashes.
> {code:java}
> ERROR Error while accepting connection (kafka.network.Acceptor)
> java.io.IOException: Too many open files in system
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> at kafka.network.Acceptor.accept(SocketServer.scala:460)
> at kafka.network.Acceptor.run(SocketServer.scala:403)
> at java.lang.Thread.run(Thread.java:748)
> {code}
>
> After some hours the issue happens again...
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7757) Too many open files after java.io.IOException: Connection to n was disconnected before the response was read
[
https://issues.apache.org/jira/browse/KAFKA-7757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pedro Gontijo updated KAFKA-7757:
-
Attachment: td2.txt
td3.txt
td1.txt
> Too many open files after java.io.IOException: Connection to n was
> disconnected before the response was read
>
>
> Key: KAFKA-7757
> URL: https://issues.apache.org/jira/browse/KAFKA-7757
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 2.1.0
>Reporter: Pedro Gontijo
>Priority: Major
> Attachments: server.properties, td1.txt, td2.txt, td3.txt
>
>
> We upgraded from 0.10.2.2 to 2.1.0 (a cluster with 3 brokers)
> After a while (hours) 2 brokers start to throw:
> {code:java}
> java.io.IOException: Connection to NN was disconnected before the response
> was read
> at
> org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:97)
> at
> kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:97)
> at
> kafka.server.ReplicaFetcherThread.fetchFromLeader(ReplicaFetcherThread.scala:190)
> at
> kafka.server.AbstractFetcherThread.kafka$server$AbstractFetcherThread$$processFetchRequest(AbstractFetcherThread.scala:241)
> at
> kafka.server.AbstractFetcherThread$$anonfun$maybeFetch$1.apply(AbstractFetcherThread.scala:130)
> at
> kafka.server.AbstractFetcherThread$$anonfun$maybeFetch$1.apply(AbstractFetcherThread.scala:129)
> at scala.Option.foreach(Option.scala:257)
> at
> kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:129)
> at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:111)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
> {code}
> The problem has happened with all brokers.
> File descriptors start to pile up and if I do not restart it throws "Too many
> open files" and crashes.
> {code:java}
> ERROR Error while accepting connection (kafka.network.Acceptor)
> java.io.IOException: Too many open files in system
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> at kafka.network.Acceptor.accept(SocketServer.scala:460)
> at kafka.network.Acceptor.run(SocketServer.scala:403)
> at java.lang.Thread.run(Thread.java:748)
> {code}
>
> After some hours the issue happens again...
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7757) Too many open files after java.io.IOException: Connection to n was disconnected before the response was read
[
https://issues.apache.org/jira/browse/KAFKA-7757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pedro Gontijo updated KAFKA-7757:
-
Description:
We upgraded from 0.10.2.2 to 2.1.0 (a cluster with 3 brokers)
After a while (hours) 2 brokers start to throw:
{code:java}
java.io.IOException: Connection to NN was disconnected before the response was
read
at
org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:97)
at
kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:97)
at
kafka.server.ReplicaFetcherThread.fetchFromLeader(ReplicaFetcherThread.scala:190)
at
kafka.server.AbstractFetcherThread.kafka$server$AbstractFetcherThread$$processFetchRequest(AbstractFetcherThread.scala:241)
at
kafka.server.AbstractFetcherThread$$anonfun$maybeFetch$1.apply(AbstractFetcherThread.scala:130)
at
kafka.server.AbstractFetcherThread$$anonfun$maybeFetch$1.apply(AbstractFetcherThread.scala:129)
at scala.Option.foreach(Option.scala:257)
at
kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:129)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:111)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
{code}
The problem has happened with all brokers.
File descriptors start to pile up and if I do not restart it throws "Too many
open files" and crashes.
{code:java}
ERROR Error while accepting connection (kafka.network.Acceptor)
java.io.IOException: Too many open files in system
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at kafka.network.Acceptor.accept(SocketServer.scala:460)
at kafka.network.Acceptor.run(SocketServer.scala:403)
at java.lang.Thread.run(Thread.java:748)
{code}
After some hours the issue happens again...
was:
We upgraded from 0.10.2.2 to 2.1.0 (a cluster with 3 brokers)
After a while (hours) 2 brokers start to throw:
{code:java}
java.io.IOException: Connection to NN was disconnected before the response was
read
at
org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:97)
at
kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:97)
at
kafka.server.ReplicaFetcherThread.fetchFromLeader(ReplicaFetcherThread.scala:190)
at
kafka.server.AbstractFetcherThread.kafka$server$AbstractFetcherThread$$processFetchRequest(AbstractFetcherThread.scala:241)
at
kafka.server.AbstractFetcherThread$$anonfun$maybeFetch$1.apply(AbstractFetcherThread.scala:130)
at
kafka.server.AbstractFetcherThread$$anonfun$maybeFetch$1.apply(AbstractFetcherThread.scala:129)
at scala.Option.foreach(Option.scala:257)
at
kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:129)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:111)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
{code}
The problem has happened with all brokers.
File descriptors start to pile up and if I do not restart it throws "Too many
open files" and crashes.
{code:java}
ERROR Error while accepting connection (kafka.network.Acceptor)
java.io.IOException: Too many open files in system
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at kafka.network.Acceptor.accept(SocketServer.scala:460)
at kafka.network.Acceptor.run(SocketServer.scala:403)
at java.lang.Thread.run(Thread.java:748)
{code}
> Too many open files after java.io.IOException: Connection to n was
> disconnected before the response was read
>
>
> Key: KAFKA-7757
> URL: https://issues.apache.org/jira/browse/KAFKA-7757
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 2.1.0
>Reporter: Pedro Gontijo
>Priority: Major
>
> We upgraded from 0.10.2.2 to 2.1.0 (a cluster with 3 brokers)
> After a while (hours) 2 brokers start to throw:
> {code:java}
> java.io.IOException: Connection to NN was disconnected before the response
> was read
> at
> org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:97)
> at
> kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:97)
> at
> kafka.server.ReplicaFetcherThread.fetchFromLeader(ReplicaFetcherThread.scala:190)
> at
> kafka.server.AbstractFetcherThread.kafka$server$AbstractFetcherThread$$processFetchRequest(AbstractFetcherThread.scala:241)
> at
> kafka.server.AbstractFetcherThread$$anonfun$maybeFetch$1.apply(AbstractFetcherTh
[jira] [Created] (KAFKA-7757) Too many open files after java.io.IOException: Connection to n was disconnected before the response was read
Pedro Gontijo created KAFKA-7757:
Summary: Too many open files after java.io.IOException: Connection
to n was disconnected before the response was read
Key: KAFKA-7757
URL: https://issues.apache.org/jira/browse/KAFKA-7757
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 2.1.0
Reporter: Pedro Gontijo
We upgraded from 0.10.2.2 to 2.1.0 (a cluster with 3 brokers)
After a while (hours) 2 brokers start to throw:
{code:java}
java.io.IOException: Connection to NN was disconnected before the response was
read
at
org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:97)
at
kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:97)
at
kafka.server.ReplicaFetcherThread.fetchFromLeader(ReplicaFetcherThread.scala:190)
at
kafka.server.AbstractFetcherThread.kafka$server$AbstractFetcherThread$$processFetchRequest(AbstractFetcherThread.scala:241)
at
kafka.server.AbstractFetcherThread$$anonfun$maybeFetch$1.apply(AbstractFetcherThread.scala:130)
at
kafka.server.AbstractFetcherThread$$anonfun$maybeFetch$1.apply(AbstractFetcherThread.scala:129)
at scala.Option.foreach(Option.scala:257)
at
kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:129)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:111)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
{code}
The problem has happened with all brokers.
File descriptors start to pile up and if I do not restart it throws "Too many
open files" and crashes.
{code:java}
ERROR Error while accepting connection (kafka.network.Acceptor)
java.io.IOException: Too many open files in system
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at kafka.network.Acceptor.accept(SocketServer.scala:460)
at kafka.network.Acceptor.run(SocketServer.scala:403)
at java.lang.Thread.run(Thread.java:748)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
