[jira] [Commented] (KAFKA-7837) maybeShrinkIsr may not reflect OfflinePartitions immediately
[
https://issues.apache.org/jira/browse/KAFKA-7837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16751766#comment-16751766
]
Jun Rao commented on KAFKA-7837:
[~dhruvilshah], the following is the description of the iterator of
ConcurrentHashmap.
"The view's {{iterator}} is a "weakly consistent" iterator that will never
throw
[{{ConcurrentModificationException}}|https://docs.oracle.com/javase/7/docs/api/java/util/ConcurrentModificationException.html],
and guarantees to traverse elements as they existed upon construction of the
iterator, and may (but is not guaranteed to) reflect any modifications
subsequent to construction."
So, it only traverses elements existed at the construction of the iterator. The
following update changes an existing element. So, I am not sure if it's
guaranteed to be reflected during the iteration.
{code:java}
allPartitions.put(topicPartition, ReplicaManager.OfflinePartition){code}
> maybeShrinkIsr may not reflect OfflinePartitions immediately
>
>
> Key: KAFKA-7837
> URL: https://issues.apache.org/jira/browse/KAFKA-7837
> Project: Kafka
> Issue Type: Improvement
>Reporter: Jun Rao
>Assignee: Dhruvil Shah
>Priority: Major
>
> When a partition is marked offline due to a failed disk, the leader is
> supposed to not shrink its ISR any more. In ReplicaManager.maybeShrinkIsr(),
> we iterate through all non-offline partitions to shrink the ISR. If an ISR
> needs to shrink, we need to write the new ISR to ZK, which can take a bit of
> time. In this window, some partitions could now be marked as offline, but may
> not be picked up by the iterator since it only reflects the state at that
> point. This can cause all in-sync followers to be dropped out of ISR
> unnecessarily and prevents a clean leader election.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7692) updateFirstUnstableOffset NPE due to sequenceId overflow in ProducerStateManager.append
[
https://issues.apache.org/jira/browse/KAFKA-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Gustafson updated KAFKA-7692:
---
Fix Version/s: 1.1.2
> updateFirstUnstableOffset NPE due to sequenceId overflow in
> ProducerStateManager.append
> ---
>
> Key: KAFKA-7692
> URL: https://issues.apache.org/jira/browse/KAFKA-7692
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 2.0.0
>Reporter: Ming Liu
>Priority: Major
> Labels: easyfix
> Fix For: 1.1.2, 2.2.0, 2.1.1, 2.0.2
>
>
> When operating Kafka, we frequently saw this exception on Kafka server log,
> {noformat}
> ERROR [KafkaServer id=19] Fatal error during KafkaServer startup. Prepare to
> shutdown (kafka.server.KafkaServer)
> java.lang.NullPointerException
> at kafka.log.Log.kafka$log$Log$$updateFirstUnstableOffset(Log.scala:792)
> at kafka.log.Log.kafka$log$Log$$loadProducerState(Log.scala:518)
> at kafka.log.Log.(Log.scala:228)
> at kafka.log.Log$.apply(Log.scala:1747)
> at
> kafka.log.LogManager.kafka$log$LogManager$$loadLog(LogManager.scala:255)
> at
> kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$11$$anonfun$apply$15$$anonfun$apply$2.apply$mcV$sp(LogManager.scala:335)
> at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:62)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7693) "IllegalArgumentException: Invalid negative sequence number used" in Kafka Client
[
https://issues.apache.org/jira/browse/KAFKA-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16751745#comment-16751745
]
ASF GitHub Bot commented on KAFKA-7693:
---
hachikuji commented on pull request #5989: KAFKA-7693: Fix SequenceNumber
overflow in client TransactionManager.
URL: https://github.com/apache/kafka/pull/5989
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> "IllegalArgumentException: Invalid negative sequence number used" in Kafka
> Client
> -
>
> Key: KAFKA-7693
> URL: https://issues.apache.org/jira/browse/KAFKA-7693
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Affects Versions: 1.1.0, 2.0.1, 2.1.0
>Reporter: Ming Liu
>Priority: Major
> Labels: easyfix
>
> When operating Kafka, we have seen Kafka client logging this kind of
> exception:
> {noformat}
> org.apache.kafka.clients.producer.internals.Sender -
> [Producer
> clientId=interaction-counter-service-2-dev-by-tweet-id-counting-dest-producer,
>
> transactionalId=interaction-counter-service-2-dev-by-tweet-id-counting-81-transactional-id-gen-1]
> Uncaught error in kafka producer I/O thread:
> java.lang.IllegalArgumentException: Invalid negative sequence number used
> at
> org.apache.kafka.common.record.MemoryRecordsBuilder.validateProducerState(MemoryRecordsBuilder.java:331)
> at
> org.apache.kafka.common.record.MemoryRecordsBuilder.close(MemoryRecordsBuilder.java:302)
> at
> org.apache.kafka.clients.producer.internals.ProducerBatch.close(ProducerBatch.java:407)
> at
> org.apache.kafka.clients.producer.internals.RecordAccumulator.drain(RecordAccumulator.java:572)
> at
> org.apache.kafka.clients.producer.internals.Sender.sendProducerData(Sender.java:270)
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:238)
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7692) updateFirstUnstableOffset NPE due to sequenceId overflow in ProducerStateManager.append
[
https://issues.apache.org/jira/browse/KAFKA-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16751740#comment-16751740
]
ASF GitHub Bot commented on KAFKA-7692:
---
hachikuji commented on pull request #5990: KAFKA-7692: Fix ProduceStateManager
SequenceNumber overflow.
URL: https://github.com/apache/kafka/pull/5990
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> updateFirstUnstableOffset NPE due to sequenceId overflow in
> ProducerStateManager.append
> ---
>
> Key: KAFKA-7692
> URL: https://issues.apache.org/jira/browse/KAFKA-7692
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 2.0.0
>Reporter: Ming Liu
>Priority: Major
> Labels: easyfix
> Fix For: 2.2.0
>
>
> When operating Kafka, we frequently saw this exception on Kafka server log,
> {noformat}
> ERROR [KafkaServer id=19] Fatal error during KafkaServer startup. Prepare to
> shutdown (kafka.server.KafkaServer)
> java.lang.NullPointerException
> at kafka.log.Log.kafka$log$Log$$updateFirstUnstableOffset(Log.scala:792)
> at kafka.log.Log.kafka$log$Log$$loadProducerState(Log.scala:518)
> at kafka.log.Log.(Log.scala:228)
> at kafka.log.Log$.apply(Log.scala:1747)
> at
> kafka.log.LogManager.kafka$log$LogManager$$loadLog(LogManager.scala:255)
> at
> kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$11$$anonfun$apply$15$$anonfun$apply$2.apply$mcV$sp(LogManager.scala:335)
> at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:62)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7866) Duplicate offsets after transaction index append failure
[ https://issues.apache.org/jira/browse/KAFKA-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson updated KAFKA-7866: --- Description: We have encountered a situation in which an ABORT marker was written successfully to the log, but failed to be written to the transaction index. This prevented the log end offset from being incremented. This resulted in duplicate offsets when the next append was attempted. The broker was using JBOD and we would normally expect IOExceptions to cause the log directory to be failed. That did not seem to happen here and the duplicates continued for several hours. Unfortunately, we are not sure what the cause of the failure was. Significantly, the first duplicate was also the first ABORT marker in the log. Unlike the offset and timestamp index, the transaction index is created on demand after the first aborted transction. It is likely that the attempt to create and open the transaction index failed. There is some suggestion that the process may have bumped into the open file limit. Whatever the problem was, it also prevented log collection, so we cannot confirm our guesses. Without knowing the underlying cause, we can still consider some potential improvements: 1. We probably should be catching non-IO exceptions in the append process. If the append to one of the indexes fails, we potentially truncate the log or re-throw it as an IOException to ensure that the log directory is no longer used. 2. Even without the unexpected exception, there is a small window during which even an IOException could lead to duplicate offsets. Marking a log directory offline is an asynchronous operation and there is no guarantee that another append cannot happen first. Given this, we probably need to detect and truncate duplicates during the log recovery process. was: We have encountered a situation in which an ABORT marker was written successfully to the log, but failed to be written to the transaction index. This prevented the log end offset from being incremented. This resulted in duplicate offsets when the next append was attempted. The broker was using JBOD and we would normally expect IOExceptions to cause the log directory to be failed. That did not seem to happen here and the duplicates continued for several hours. Unfortunately, we are not sure what the cause of the failure was. Significantly, the first duplicate was also the first ABORT marker in the log. Unlike the offset and timestamp index, the transaction index is created on demand after the first aborted transction. It is likely that the attempt to create and open the transaction index failed. There is some suggestion that the process may have bumped into the open file limit. Whatever the problem was, it also prevented log collection, so we cannot Without knowing the underlying cause, we can still consider some potential improvements: 1. We probably should be catching non-IO exceptions in the append process. If the append to one of the indexes fails, we potentially truncate the log or re-throw it as an IOException to ensure that the log directory is no longer used. 2. Even without the unexpected exception, there is a small window during which even an IOException could lead to duplicate offsets. Marking a log directory offline is an asynchronous operation and there is no guarantee that another append cannot happen first. Given this, we probably need to detect and truncate duplicates during the log recovery process. > Duplicate offsets after transaction index append failure > > > Key: KAFKA-7866 > URL: https://issues.apache.org/jira/browse/KAFKA-7866 > Project: Kafka > Issue Type: Bug >Reporter: Jason Gustafson >Priority: Major > > We have encountered a situation in which an ABORT marker was written > successfully to the log, but failed to be written to the transaction index. > This prevented the log end offset from being incremented. This resulted in > duplicate offsets when the next append was attempted. The broker was using > JBOD and we would normally expect IOExceptions to cause the log directory to > be failed. That did not seem to happen here and the duplicates continued for > several hours. > Unfortunately, we are not sure what the cause of the failure was. > Significantly, the first duplicate was also the first ABORT marker in the > log. Unlike the offset and timestamp index, the transaction index is created > on demand after the first aborted transction. It is likely that the attempt > to create and open the transaction index failed. There is some suggestion > that the process may have bumped into the open file limit. Whatever the > problem was, it also prevented log collection, so we cannot confirm our > guesses. > Without knowing the underly
[jira] [Comment Edited] (KAFKA-7741) Bad dependency via SBT
[
https://issues.apache.org/jira/browse/KAFKA-7741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16751680#comment-16751680
]
Colin P. McCabe edited comment on KAFKA-7741 at 1/24/19 11:13 PM:
--
bq. The fix is merged, so I'll update the fix versions. I still need to update
the documentation with a workaround for the versions that are already released.
Hi John,
Thanks for working on this. The fix has been merged, so I think it's better
for the JIRA to be closed. If you want to implement a different fix for some
older Kafka versions, it's probably better to open a separate JIRA for that and
link it here.
Otherwise, it creates confusion. For example, people might want to know if
this is fixed in 2.1.1 (which it is). But if they look at this JIRA and see
that it is open, people will get the impression that it is not fixed.
was (Author: cmccabe):
bq. The fix is merged, so I'll update the fix versions. I still need to update
the documentation with a workaround for the versions that are already released.
bq.
Hi John,
Thanks for working on this. The fix has been merged, so the JIRA should
properly be considered closed. If you want to implement a different fix for
some older Kafka versions, please open a separate JIRA for that and link it
here.
Otherwise, it creates confusion. For example, people might want to know if
this is fixed in 2.1.1. It is, of course. But if they look at this JIRA and
see that it is open, people will get the impression that it is not fixed.
> Bad dependency via SBT
> --
>
> Key: KAFKA-7741
> URL: https://issues.apache.org/jira/browse/KAFKA-7741
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.0.0, 2.0.1, 2.1.0
> Environment: Windows 10 professional, IntelliJ IDEA 2017.1
>Reporter: sacha barber
>Assignee: John Roesler
>Priority: Major
> Fix For: 2.2.0, 2.1.1, 2.0.2
>
>
> I am using the Kafka-Streams-Scala 2.1.0 JAR.
> And if I create a new Scala project using SBT with these dependencies
> {code}
> name := "ScalaKafkaStreamsDemo"
> version := "1.0"
> scalaVersion := "2.12.1"
> libraryDependencies += "org.apache.kafka" %% "kafka" % "2.0.0"
> libraryDependencies += "org.apache.kafka" % "kafka-clients" % "2.0.0"
> libraryDependencies += "org.apache.kafka" % "kafka-streams" % "2.0.0"
> libraryDependencies += "org.apache.kafka" %% "kafka-streams-scala" % "2.0.0"
> //TEST
> libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.5" % Test
> libraryDependencies += "org.apache.kafka" % "kafka-streams-test-utils" %
> "2.0.0" % Test
> {code}
> I get this error
>
> {code}
> SBT 'ScalaKafkaStreamsDemo' project refresh failed
> Error:Error while importing SBT project:...[info] Resolving
> jline#jline;2.14.1 ...
> [warn] [FAILED ]
> javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}: (0ms)
> [warn] local: tried
> [warn]
> C:\Users\sacha\.ivy2\local\javax.ws.rs\javax.ws.rs-api\2.1.1\${packaging.type}s\javax.ws.rs-api.${packaging.type}
> [warn] public: tried
> [warn]
> https://repo1.maven.org/maven2/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.${packaging.type}
> [info] downloading
> https://repo1.maven.org/maven2/org/apache/kafka/kafka-streams-test-utils/2.1.0/kafka-streams-test-utils-2.1.0.jar
> ...
> [info] [SUCCESSFUL ]
> org.apache.kafka#kafka-streams-test-utils;2.1.0!kafka-streams-test-utils.jar
> (344ms)
> [warn] ::
> [warn] :: FAILED DOWNLOADS ::
> [warn] :: ^ see resolution messages for details ^ ::
> [warn] ::
> [warn] :: javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}
> [warn] ::
> [trace] Stack trace suppressed: run 'last *:ssExtractDependencies' for the
> full output.
> [trace] Stack trace suppressed: run 'last *:update' for the full output.
> [error] (*:ssExtractDependencies) sbt.ResolveException: download failed:
> javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}
> [error] (*:update) sbt.ResolveException: download failed:
> javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}
> [error] Total time: 8 s, completed 16-Dec-2018 19:27:21
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=384M;
> support was removed in 8.0See complete log in href="file:/C:/Users/sacha/.IdeaIC2017.1/system/log/sbt.last.log">file:/C:/Users/sacha/.IdeaIC2017.1/system/log/sbt.last.log
> {code}
> This seems to be a common issue with bad dependency from Kafka to
> javax.ws.rs-api.
> if I drop the Kafka version down to 2.0.0 and add this line to my SBT file
> this error goes away
> {code}
> libraryDependencies += "javax.ws.rs" % "javax.ws.rs-api"
[jira] [Updated] (KAFKA-7856) Cryptographic Issues by Insufficient Entropy
[ https://issues.apache.org/jira/browse/KAFKA-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin P. McCabe updated KAFKA-7856: --- Affects Version/s: (was: 2.1.0) Fix Version/s: (was: 2.1.1) > Cryptographic Issues by Insufficient Entropy > > > Key: KAFKA-7856 > URL: https://issues.apache.org/jira/browse/KAFKA-7856 > Project: Kafka > Issue Type: Bug >Reporter: Victor Sartori >Priority: Major > Labels: patch, pull-request-available, security > > We pass the kakfa client in security analisys ans this scans reports: > CWE-331 - Flaw medium,SANS TOP 25 > [https://cwe.mitre.org/data/definitions/331.html] > > A PR on github is present. (https://github.com/apache/kafka/pull/6184) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-2145) An option to add topic owners.
[ https://issues.apache.org/jira/browse/KAFKA-2145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16751676#comment-16751676 ] ASF GitHub Bot commented on KAFKA-2145: --- Parth-Brahmbhatt commented on pull request #77: KAFKA-2145: Add a log config so users can define topic owners. URL: https://github.com/apache/kafka/pull/77 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > An option to add topic owners. > --- > > Key: KAFKA-2145 > URL: https://issues.apache.org/jira/browse/KAFKA-2145 > Project: Kafka > Issue Type: Improvement >Reporter: Parth Brahmbhatt >Assignee: Parth Brahmbhatt >Priority: Major > > We need to expose a way so users can identify users/groups that share > ownership of topic. We discussed adding this as part of > https://issues.apache.org/jira/browse/KAFKA-2035 and agreed that it will be > simpler to add owner as a logconfig. > The owner field can be used for auditing and also by authorization layer to > grant access without having to explicitly configure acls. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-7741) Bad dependency via SBT
[
https://issues.apache.org/jira/browse/KAFKA-7741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin P. McCabe resolved KAFKA-7741.
Resolution: Fixed
> Bad dependency via SBT
> --
>
> Key: KAFKA-7741
> URL: https://issues.apache.org/jira/browse/KAFKA-7741
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.0.0, 2.0.1, 2.1.0
> Environment: Windows 10 professional, IntelliJ IDEA 2017.1
>Reporter: sacha barber
>Assignee: John Roesler
>Priority: Major
> Fix For: 2.2.0, 2.1.1, 2.0.2
>
>
> I am using the Kafka-Streams-Scala 2.1.0 JAR.
> And if I create a new Scala project using SBT with these dependencies
> {code}
> name := "ScalaKafkaStreamsDemo"
> version := "1.0"
> scalaVersion := "2.12.1"
> libraryDependencies += "org.apache.kafka" %% "kafka" % "2.0.0"
> libraryDependencies += "org.apache.kafka" % "kafka-clients" % "2.0.0"
> libraryDependencies += "org.apache.kafka" % "kafka-streams" % "2.0.0"
> libraryDependencies += "org.apache.kafka" %% "kafka-streams-scala" % "2.0.0"
> //TEST
> libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.5" % Test
> libraryDependencies += "org.apache.kafka" % "kafka-streams-test-utils" %
> "2.0.0" % Test
> {code}
> I get this error
>
> {code}
> SBT 'ScalaKafkaStreamsDemo' project refresh failed
> Error:Error while importing SBT project:...[info] Resolving
> jline#jline;2.14.1 ...
> [warn] [FAILED ]
> javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}: (0ms)
> [warn] local: tried
> [warn]
> C:\Users\sacha\.ivy2\local\javax.ws.rs\javax.ws.rs-api\2.1.1\${packaging.type}s\javax.ws.rs-api.${packaging.type}
> [warn] public: tried
> [warn]
> https://repo1.maven.org/maven2/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.${packaging.type}
> [info] downloading
> https://repo1.maven.org/maven2/org/apache/kafka/kafka-streams-test-utils/2.1.0/kafka-streams-test-utils-2.1.0.jar
> ...
> [info] [SUCCESSFUL ]
> org.apache.kafka#kafka-streams-test-utils;2.1.0!kafka-streams-test-utils.jar
> (344ms)
> [warn] ::
> [warn] :: FAILED DOWNLOADS ::
> [warn] :: ^ see resolution messages for details ^ ::
> [warn] ::
> [warn] :: javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}
> [warn] ::
> [trace] Stack trace suppressed: run 'last *:ssExtractDependencies' for the
> full output.
> [trace] Stack trace suppressed: run 'last *:update' for the full output.
> [error] (*:ssExtractDependencies) sbt.ResolveException: download failed:
> javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}
> [error] (*:update) sbt.ResolveException: download failed:
> javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}
> [error] Total time: 8 s, completed 16-Dec-2018 19:27:21
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=384M;
> support was removed in 8.0See complete log in href="file:/C:/Users/sacha/.IdeaIC2017.1/system/log/sbt.last.log">file:/C:/Users/sacha/.IdeaIC2017.1/system/log/sbt.last.log
> {code}
> This seems to be a common issue with bad dependency from Kafka to
> javax.ws.rs-api.
> if I drop the Kafka version down to 2.0.0 and add this line to my SBT file
> this error goes away
> {code}
> libraryDependencies += "javax.ws.rs" % "javax.ws.rs-api" % "2.1"
> artifacts(Artifact("javax.ws.rs-api", "jar", "jar"))`
> {code}
>
> However I would like to work with 2.1.0 version.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7741) Bad dependency via SBT
[
https://issues.apache.org/jira/browse/KAFKA-7741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16751680#comment-16751680
]
Colin P. McCabe commented on KAFKA-7741:
bq. The fix is merged, so I'll update the fix versions. I still need to update
the documentation with a workaround for the versions that are already released.
bq.
Hi John,
Thanks for working on this. The fix has been merged, so the JIRA should
properly be considered closed. If you want to implement a different fix for
some older Kafka versions, please open a separate JIRA for that and link it
here.
Otherwise, it creates confusion. For example, people might want to know if
this is fixed in 2.1.1. It is, of course. But if they look at this JIRA and
see that it is open, people will get the impression that it is not fixed.
> Bad dependency via SBT
> --
>
> Key: KAFKA-7741
> URL: https://issues.apache.org/jira/browse/KAFKA-7741
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.0.0, 2.0.1, 2.1.0
> Environment: Windows 10 professional, IntelliJ IDEA 2017.1
>Reporter: sacha barber
>Assignee: John Roesler
>Priority: Major
> Fix For: 2.2.0, 2.1.1, 2.0.2
>
>
> I am using the Kafka-Streams-Scala 2.1.0 JAR.
> And if I create a new Scala project using SBT with these dependencies
> {code}
> name := "ScalaKafkaStreamsDemo"
> version := "1.0"
> scalaVersion := "2.12.1"
> libraryDependencies += "org.apache.kafka" %% "kafka" % "2.0.0"
> libraryDependencies += "org.apache.kafka" % "kafka-clients" % "2.0.0"
> libraryDependencies += "org.apache.kafka" % "kafka-streams" % "2.0.0"
> libraryDependencies += "org.apache.kafka" %% "kafka-streams-scala" % "2.0.0"
> //TEST
> libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.5" % Test
> libraryDependencies += "org.apache.kafka" % "kafka-streams-test-utils" %
> "2.0.0" % Test
> {code}
> I get this error
>
> {code}
> SBT 'ScalaKafkaStreamsDemo' project refresh failed
> Error:Error while importing SBT project:...[info] Resolving
> jline#jline;2.14.1 ...
> [warn] [FAILED ]
> javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}: (0ms)
> [warn] local: tried
> [warn]
> C:\Users\sacha\.ivy2\local\javax.ws.rs\javax.ws.rs-api\2.1.1\${packaging.type}s\javax.ws.rs-api.${packaging.type}
> [warn] public: tried
> [warn]
> https://repo1.maven.org/maven2/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.${packaging.type}
> [info] downloading
> https://repo1.maven.org/maven2/org/apache/kafka/kafka-streams-test-utils/2.1.0/kafka-streams-test-utils-2.1.0.jar
> ...
> [info] [SUCCESSFUL ]
> org.apache.kafka#kafka-streams-test-utils;2.1.0!kafka-streams-test-utils.jar
> (344ms)
> [warn] ::
> [warn] :: FAILED DOWNLOADS ::
> [warn] :: ^ see resolution messages for details ^ ::
> [warn] ::
> [warn] :: javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}
> [warn] ::
> [trace] Stack trace suppressed: run 'last *:ssExtractDependencies' for the
> full output.
> [trace] Stack trace suppressed: run 'last *:update' for the full output.
> [error] (*:ssExtractDependencies) sbt.ResolveException: download failed:
> javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}
> [error] (*:update) sbt.ResolveException: download failed:
> javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}
> [error] Total time: 8 s, completed 16-Dec-2018 19:27:21
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=384M;
> support was removed in 8.0See complete log in href="file:/C:/Users/sacha/.IdeaIC2017.1/system/log/sbt.last.log">file:/C:/Users/sacha/.IdeaIC2017.1/system/log/sbt.last.log
> {code}
> This seems to be a common issue with bad dependency from Kafka to
> javax.ws.rs-api.
> if I drop the Kafka version down to 2.0.0 and add this line to my SBT file
> this error goes away
> {code}
> libraryDependencies += "javax.ws.rs" % "javax.ws.rs-api" % "2.1"
> artifacts(Artifact("javax.ws.rs-api", "jar", "jar"))`
> {code}
>
> However I would like to work with 2.1.0 version.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-2854) Make KerberosName implement PrincipalToLocal plugin so authorizer and authenticator can share this.
[ https://issues.apache.org/jira/browse/KAFKA-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16751675#comment-16751675 ] ASF GitHub Bot commented on KAFKA-2854: --- Parth-Brahmbhatt commented on pull request #547: KAFKA-2854: Making KerberosShortNamer implement an interface and making it pluggable. URL: https://github.com/apache/kafka/pull/547 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Make KerberosName implement PrincipalToLocal plugin so authorizer and > authenticator can share this. > --- > > Key: KAFKA-2854 > URL: https://issues.apache.org/jira/browse/KAFKA-2854 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.9.0.0 >Reporter: Parth Brahmbhatt >Assignee: Parth Brahmbhatt >Priority: Major > > Both Authorizer and SASL Authenticator needs a way to convert kerberos > principal into a local identity. This jira proposes to add an interface and a > config for this conversion so users can inject their own implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7693) "IllegalArgumentException: Invalid negative sequence number used" in Kafka Client
[
https://issues.apache.org/jira/browse/KAFKA-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin P. McCabe updated KAFKA-7693:
---
Fix Version/s: (was: 2.1.1)
> "IllegalArgumentException: Invalid negative sequence number used" in Kafka
> Client
> -
>
> Key: KAFKA-7693
> URL: https://issues.apache.org/jira/browse/KAFKA-7693
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Affects Versions: 1.1.0, 2.0.1, 2.1.0
>Reporter: Ming Liu
>Priority: Major
> Labels: easyfix
>
> When operating Kafka, we have seen Kafka client logging this kind of
> exception:
> {noformat}
> org.apache.kafka.clients.producer.internals.Sender -
> [Producer
> clientId=interaction-counter-service-2-dev-by-tweet-id-counting-dest-producer,
>
> transactionalId=interaction-counter-service-2-dev-by-tweet-id-counting-81-transactional-id-gen-1]
> Uncaught error in kafka producer I/O thread:
> java.lang.IllegalArgumentException: Invalid negative sequence number used
> at
> org.apache.kafka.common.record.MemoryRecordsBuilder.validateProducerState(MemoryRecordsBuilder.java:331)
> at
> org.apache.kafka.common.record.MemoryRecordsBuilder.close(MemoryRecordsBuilder.java:302)
> at
> org.apache.kafka.clients.producer.internals.ProducerBatch.close(ProducerBatch.java:407)
> at
> org.apache.kafka.clients.producer.internals.RecordAccumulator.drain(RecordAccumulator.java:572)
> at
> org.apache.kafka.clients.producer.internals.Sender.sendProducerData(Sender.java:270)
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:238)
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7715) Connect should have a parameter to disable WADL output for OPTIONS method
[
https://issues.apache.org/jira/browse/KAFKA-7715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin P. McCabe updated KAFKA-7715:
---
Fix Version/s: (was: 2.1.1)
> Connect should have a parameter to disable WADL output for OPTIONS method
> -
>
> Key: KAFKA-7715
> URL: https://issues.apache.org/jira/browse/KAFKA-7715
> Project: Kafka
> Issue Type: Improvement
> Components: config, security
>Affects Versions: 2.1.0
>Reporter: Oleksandr Diachenko
>Assignee: Oleksandr Diachenko
>Priority: Critical
>
> Currently, Connect REST API exposes WADL output on OPTIONS method:
> {code:bash}
> curl -i -X OPTIONS http://localhost:8083/connectors
> HTTP/1.1 200 OK
> Date: Fri, 07 Dec 2018 22:51:53 GMT
> Content-Type: application/vnd.sun.wadl+xml
> Allow: HEAD,POST,GET,OPTIONS
> Last-Modified: Fri, 07 Dec 2018 14:51:53 PST
> Content-Length: 1331
> Server: Jetty(9.4.12.v20180830)
>
> http://wadl.dev.java.net/2009/02";>
> http://jersey.java.net/"; jersey:generatedBy="Jersey: 2.27
> 2018-04-10 07:34:57"/>
>
> http://localhost:8083/application.wadl/xsd0.xsd";>
>
>
>
> http://localhost:8083/";>
>
>
>
> http://www.w3.org/2001/XMLSchema"; name="forward"
> style="query" type="xs:boolean"/>
>
>
>
>
>
>
>
>
> http://www.w3.org/2001/XMLSchema"; name="forward"
> style="query" type="xs:boolean"/>
>
>
>
>
>
>
>
>
> {code}
> This can be a potential vulnerability, so it makes sense to have a
> configuration parameter, which disables WADL output.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7672) The local state not fully restored after KafkaStream rebalanced, resulting in data loss
[
https://issues.apache.org/jira/browse/KAFKA-7672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin P. McCabe updated KAFKA-7672:
---
Fix Version/s: (was: 2.1.1)
> The local state not fully restored after KafkaStream rebalanced, resulting in
> data loss
> ---
>
> Key: KAFKA-7672
> URL: https://issues.apache.org/jira/browse/KAFKA-7672
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 1.1.0, 1.1.1, 2.0.0, 2.1.0
>Reporter: linyue li
>Assignee: linyue li
>Priority: Major
> Labels: bug
> Fix For: 2.2.0
>
>
> Normally, when a task is migrated to a new thread and no checkpoint file was
> found under its task folder, Kafka Stream needs to restore the local state
> for remote changelog topic completely and then resume running. However, in
> some scenarios, we found that Kafka Stream *NOT* restore this state even no
> checkpoint was found, but just clean the state folder and transition to
> running state directly, resulting the historic data loss.
> To be specific, I will give the detailed logs for Kafka Stream in our project
> to show this scenario:
> {quote}2018-10-23 08:27:07,684 INFO
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer
> clientId=AuditTrailBatch-StreamThread-1-consumer, groupId=AuditTrailBatch]
> Revoking previously assigned partitions [AuditTrailBatch-0-5]
> 2018-10-23 08:27:07,684 INFO
> org.apache.kafka.streams.processor.internals.StreamThread - stream-thread
> [AuditTrailBatch-StreamThread-1] State transition from PARTITIONS_ASSIGNED to
> PARTITIONS_REVOKED
> 2018-10-23 08:27:10,856 INFO
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer
> clientId=AuditTrailBatch-StreamThread-1-consumer, groupId=AuditTrailBatch]
> (Re-)joining group
> 2018-10-23 08:27:53,153 INFO
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer
> clientId=AuditTrailBatch-StreamThread-1-consumer, groupId=AuditTrailBatch]
> Successfully joined group with generation 323
> 2018-10-23 08:27:53,153 INFO
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer
> clientId=AuditTrailBatch-StreamThread-1-consumer, groupId=AuditTrailBatch]
> Setting newly assigned partitions [AuditTrailBatch-store1-repartition-1]
> 2018-10-23 08:27:53,153 INFO
> org.apache.kafka.streams.processor.internals.StreamThread - stream-thread
> [AuditTrailBatch-StreamThread-1] State transition from PARTITIONS_REVOKED to
> PARTITIONS_ASSIGNED
> 2018-10-23 08:27:53,153 INFO
> org.apache.kafka.streams.processor.internals.StreamThread - stream-thread
> [AuditTrailBatch-StreamThread-1] *Creating producer client for task 1_1*
> 2018-10-23 08:27:53,622 INFO
> org.apache.kafka.streams.processor.internals.StreamThread - stream-thread
> [AuditTrailBatch-StreamThread-1] partition assignment took 469 ms.
> 2018-10-23 08:27:54,357 INFO
> org.apache.kafka.streams.processor.internals.StoreChangelogReader -
> stream-thread [AuditTrailBatch-StreamThread-1]*No checkpoint found for task
> 1_1 state store AuditTrailBatch-store1-changelog-1 with EOS turned on.*
> *Reinitializing the task and restore its state from the beginning.*
> 2018-10-23 08:27:54,357 INFO
> org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer
> clientId=AuditTrailBatch-StreamThread-1-restore-consumer, groupId=]*Resetting
> offset for partition AuditTrailBatch-store1-changelog-1 to offset 0.*
> 2018-10-23 08:27:54,653 INFO
> org.apache.kafka.streams.processor.internals.StreamThread - stream-thread
> [AuditTrailBatch-StreamThread-1]*State transition from PARTITIONS_ASSIGNED to
> RUNNING*
> {quote}
> From the logs above, we can get the procedure for thread
> AuditTrailBatch-StreamThread-1:
> # the previous running task assigned to thread 1 is task 0_5 (the
> corresponding partition is AuditTrailBatch-0-5)
> # group begins to rebalance, the new task 1_1 is assigned to thread 1.
> # no checkpoint was found under 1_1 state folder, so reset the offset to 0
> and clean the local state folder.
> # thread 1 transitions to RUNNING state directly without the restoration for
> task 1_1, so the historic data for state 1_1 is lost for thread 1.
> *ThoubleShoot*
> To investigate the cause for this issue, we analysis the source code in
> KafkaStream and found the key is the variable named "completedRestorers".
> This is the definition of the variable:
> {code:java}
> private final Set completedRestorers = new HashSet<>();{code}
> Each thread object has its own completedRestorers, which is created in the
> thread initialization, and not accessed crossly by other threads. The
> completedRestorers is used to record the part
[jira] [Updated] (KAFKA-7531) NPE NullPointerException at TransactionCoordinator handleEndTransaction
[
https://issues.apache.org/jira/browse/KAFKA-7531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin P. McCabe updated KAFKA-7531:
---
Fix Version/s: (was: 2.1.1)
> NPE NullPointerException at TransactionCoordinator handleEndTransaction
> ---
>
> Key: KAFKA-7531
> URL: https://issues.apache.org/jira/browse/KAFKA-7531
> Project: Kafka
> Issue Type: Bug
> Components: controller
>Affects Versions: 2.0.0
>Reporter: Sebastian Puzoń
>Priority: Critical
> Fix For: 2.0.2
>
> Attachments: server.log.2018-11-29-16.gz,
> sswvlp6801.server.log.2018-12-01-11.gz
>
>
> Kafka cluster with 4 brokers, 1 topic (20 partitions), 1 zookeeper.
> Streams Application 4 instances, each has 5 Streams threads, total 20 stream
> threads.
> I observe NPE NullPointerException at coordinator broker which causes all
> application stream threads shutdown, here's stack from broker:
> {code:java}
> [2018-10-22 21:50:46,755] INFO [GroupCoordinator 2]: Member
> elog_agg-client-sswvlp6802-StreamThread-4-consumer-cbcd4704-a346-45ea-80f9-96f62fc2dabe
> in group elo
> g_agg has failed, removing it from the group
> (kafka.coordinator.group.GroupCoordinator)
> [2018-10-22 21:50:46,755] INFO [GroupCoordinator 2]: Preparing to rebalance
> group elog_agg with old generation 49 (__consumer_offsets-21)
> (kafka.coordinator.gro
> up.GroupCoordinator)
> [2018-10-22 21:51:17,519] INFO [GroupCoordinator 2]: Stabilized group
> elog_agg generation 50 (__consumer_offsets-21)
> (kafka.coordinator.group.GroupCoordinator)
> [2018-10-22 21:51:17,524] INFO [GroupCoordinator 2]: Assignment received from
> leader for group elog_agg for generation 50
> (kafka.coordinator.group.GroupCoordina
> tor)
> [2018-10-22 21:51:27,596] INFO [TransactionCoordinator id=2] Initialized
> transactionalId elog_agg-0_14 with producerId 1001 and producer epoch 20 on
> partition _
> _transaction_state-16 (kafka.coordinator.transaction.TransactionCoordinator)
> [
> [2018-10-22 21:52:00,920] ERROR [KafkaApi-2] Error when handling request
> {transactional_id=elog_agg-0_3,producer_id=1004,producer_epoch=16,transaction_result=tr
> ue} (kafka.server.KafkaApis)
> java.lang.NullPointerException
> at
> kafka.coordinator.transaction.TransactionCoordinator.$anonfun$handleEndTransaction$7(TransactionCoordinator.scala:393)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
> at
> kafka.coordinator.transaction.TransactionMetadata.inLock(TransactionMetadata.scala:172)
> at
> kafka.coordinator.transaction.TransactionCoordinator.$anonfun$handleEndTransaction$5(TransactionCoordinator.scala:383)
> at scala.util.Either$RightProjection.flatMap(Either.scala:702)
> at
> kafka.coordinator.transaction.TransactionCoordinator.sendTxnMarkersCallback$1(TransactionCoordinator.scala:372)
> at
> kafka.coordinator.transaction.TransactionCoordinator.$anonfun$handleEndTransaction$12(TransactionCoordinator.scala:437)
> at
> kafka.coordinator.transaction.TransactionCoordinator.$anonfun$handleEndTransaction$12$adapted(TransactionCoordinator.scala:437)
> at
> kafka.coordinator.transaction.TransactionStateManager.updateCacheCallback$1(TransactionStateManager.scala:581)
> at
> kafka.coordinator.transaction.TransactionStateManager.$anonfun$appendTransactionToLog$15(TransactionStateManager.scala:619)
> at
> kafka.coordinator.transaction.TransactionStateManager.$anonfun$appendTransactionToLog$15$adapted(TransactionStateManager.scala:619)
> at kafka.server.DelayedProduce.onComplete(DelayedProduce.scala:129)
> at kafka.server.DelayedOperation.forceComplete(DelayedOperation.scala:70)
> at kafka.server.DelayedProduce.tryComplete(DelayedProduce.scala:110)
> at
> kafka.server.DelayedOperationPurgatory.tryCompleteElseWatch(DelayedOperation.scala:232)
> at kafka.server.ReplicaManager.appendRecords(ReplicaManager.scala:495)
> at
> kafka.coordinator.transaction.TransactionStateManager.$anonfun$appendTransactionToLog$13(TransactionStateManager.scala:613)
> at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
> at kafka.utils.CoreUtils$.inReadLock(CoreUtils.scala:257)
> at
> kafka.coordinator.transaction.TransactionStateManager.appendTransactionToLog(TransactionStateManager.scala:590)
> at
> kafka.coordinator.transaction.TransactionCoordinator.handleEndTransaction(TransactionCoordinator.scala:437)
> at kafka.server.KafkaApis.handleEndTxnRequest(KafkaApis.scala:1653)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:132)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:69)
> at java.lang.Thread.run(Thread.java:745)
> [2018-10-22 21:52:15,958] ERROR [KafkaApi-2] Error when handling request
> {transactional_id=elog_agg-0_9,producer_id=1005,prod
[jira] [Updated] (KAFKA-7045) Consumer may not be able to consume all messages when down-conversion is required
[ https://issues.apache.org/jira/browse/KAFKA-7045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin P. McCabe updated KAFKA-7045: --- Fix Version/s: (was: 2.1.1) > Consumer may not be able to consume all messages when down-conversion is > required > - > > Key: KAFKA-7045 > URL: https://issues.apache.org/jira/browse/KAFKA-7045 > Project: Kafka > Issue Type: Bug > Components: consumer, core >Affects Versions: 0.11.0.0, 0.11.0.1, 0.11.0.2, 1.0.0, 1.0.1, 1.1.0, 2.0.0 >Reporter: Dhruvil Shah >Priority: Major > Fix For: 2.2.0 > > Attachments: log-cleaner-test.zip > > > When down-conversion is required, the consumer might fail consuming messages > under certain conditions. Couple such cases are outlined below: > (1) When consuming from a compacted topic, it is possible that the consumer > wants to fetch messages that fall in the middle of a batch but the messages > have been compacted by the cleaner. For example, let's say we have the > following two segments. The brackets indicate a single batch of messages and > the numbers within are the message offsets. > Segment #1: [0, 1, 2], [3, 4, 5], [6, 7, 8] > Segment #2: [9, 10, 11], [12, 13, 14] > If the cleaner were to come in now and clean up messages with offsets 7 and > 8, the segments would look like the following: > Segment #1: [0, 1, 2], [3, 4, 5], [6] > Segment #2: [9, 10, 11], [12, 13, 14] > A consumer attempting to fetch messages at offset 7 will start reading the > batch starting at offset 6. During down-conversion, we will drop the record > starting at 6 it is less than the current fetch start offset. However, there > are no messages in the log following offset 6. In such cases, we return the > `FileRecords` itself which would cause the consumer to throw an exception > because it does not understand the stored message format. > (2) When consuming from a topic with transactional messages, down-conversion > usually drops control batches because these do not exist in V0 and V1 message > formats. If there are no message batches following the control batch in the > particular segment (or if we are at the end of the log), we would again get > no records after down-conversion and will return the `FileRecords`. Because > the consumer is not able to interpret control batches, it will again throw an > exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7304) memory leakage in org.apache.kafka.common.network.Selector
[
https://issues.apache.org/jira/browse/KAFKA-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin P. McCabe updated KAFKA-7304:
---
Fix Version/s: (was: 2.1.1)
> memory leakage in org.apache.kafka.common.network.Selector
> --
>
> Key: KAFKA-7304
> URL: https://issues.apache.org/jira/browse/KAFKA-7304
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 1.1.0, 1.1.1
>Reporter: Yu Yang
>Priority: Critical
> Fix For: 1.1.2, 2.2.0, 2.0.2
>
> Attachments: 7304.v4.txt, 7304.v7.txt, Screen Shot 2018-08-16 at
> 11.04.16 PM.png, Screen Shot 2018-08-16 at 11.06.38 PM.png, Screen Shot
> 2018-08-16 at 12.41.26 PM.png, Screen Shot 2018-08-16 at 4.26.19 PM.png,
> Screen Shot 2018-08-17 at 1.03.35 AM.png, Screen Shot 2018-08-17 at 1.04.32
> AM.png, Screen Shot 2018-08-17 at 1.05.30 AM.png, Screen Shot 2018-08-28 at
> 11.09.45 AM.png, Screen Shot 2018-08-29 at 10.49.03 AM.png, Screen Shot
> 2018-08-29 at 10.50.47 AM.png, Screen Shot 2018-09-29 at 10.38.12 PM.png,
> Screen Shot 2018-09-29 at 10.38.38 PM.png, Screen Shot 2018-09-29 at 8.34.50
> PM.png
>
>
> We are testing secured writing to kafka through ssl. Testing at small scale,
> ssl writing to kafka was fine. However, when we enabled ssl writing at a
> larger scale (>40k clients write concurrently), the kafka brokers soon hit
> OutOfMemory issue with 4G memory setting. We have tried with increasing the
> heap size to 10Gb, but encountered the same issue.
> We took a few heap dumps , and found that most of the heap memory is
> referenced through org.apache.kafka.common.network.Selector objects. There
> are two Channel maps field in Selector. It seems that somehow the objects is
> not deleted from the map in a timely manner.
> One observation is that the memory leak seems relate to kafka partition
> leader changes. If there is broker restart etc. in the cluster that caused
> partition leadership change, the brokers may hit the OOM issue faster.
> {code}
> private final Map channels;
> private final Map closingChannels;
> {code}
> Please see the attached images and the following link for sample gc
> analysis.
> http://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTgvMDgvMTcvLS1nYy5sb2cuMC5jdXJyZW50Lmd6LS0yLTM5LTM0
> the command line for running kafka:
> {code}
> java -Xms10g -Xmx10g -XX:NewSize=512m -XX:MaxNewSize=512m
> -Xbootclasspath/p:/usr/local/libs/bcp -XX:MetaspaceSize=128m -XX:+UseG1GC
> -XX:MaxGCPauseMillis=25 -XX:InitiatingHeapOccupancyPercent=35
> -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=25
> -XX:MaxMetaspaceFreeRatio=75 -XX:+PrintGCDetails -XX:+PrintGCDateStamps
> -XX:+PrintTenuringDistribution -Xloggc:/var/log/kafka/gc.log
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=40 -XX:GCLogFileSize=50M
> -Djava.awt.headless=true
> -Dlog4j.configuration=file:/etc/kafka/log4j.properties
> -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.port=
> -Dcom.sun.management.jmxremote.rmi.port= -cp /usr/local/libs/*
> kafka.Kafka /etc/kafka/server.properties
> {code}
> We use java 1.8.0_102, and has applied a TLS patch on reducing
> X509Factory.certCache map size from 750 to 20.
> {code}
> java -version
> java version "1.8.0_102"
> Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
> Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7741) Bad dependency via SBT
[
https://issues.apache.org/jira/browse/KAFKA-7741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin P. McCabe updated KAFKA-7741:
---
Fix Version/s: 2.1.1
> Bad dependency via SBT
> --
>
> Key: KAFKA-7741
> URL: https://issues.apache.org/jira/browse/KAFKA-7741
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.0.0, 2.0.1, 2.1.0
> Environment: Windows 10 professional, IntelliJ IDEA 2017.1
>Reporter: sacha barber
>Assignee: John Roesler
>Priority: Major
> Fix For: 2.2.0, 2.1.1, 2.0.2
>
>
> I am using the Kafka-Streams-Scala 2.1.0 JAR.
> And if I create a new Scala project using SBT with these dependencies
> {code}
> name := "ScalaKafkaStreamsDemo"
> version := "1.0"
> scalaVersion := "2.12.1"
> libraryDependencies += "org.apache.kafka" %% "kafka" % "2.0.0"
> libraryDependencies += "org.apache.kafka" % "kafka-clients" % "2.0.0"
> libraryDependencies += "org.apache.kafka" % "kafka-streams" % "2.0.0"
> libraryDependencies += "org.apache.kafka" %% "kafka-streams-scala" % "2.0.0"
> //TEST
> libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.5" % Test
> libraryDependencies += "org.apache.kafka" % "kafka-streams-test-utils" %
> "2.0.0" % Test
> {code}
> I get this error
>
> {code}
> SBT 'ScalaKafkaStreamsDemo' project refresh failed
> Error:Error while importing SBT project:...[info] Resolving
> jline#jline;2.14.1 ...
> [warn] [FAILED ]
> javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}: (0ms)
> [warn] local: tried
> [warn]
> C:\Users\sacha\.ivy2\local\javax.ws.rs\javax.ws.rs-api\2.1.1\${packaging.type}s\javax.ws.rs-api.${packaging.type}
> [warn] public: tried
> [warn]
> https://repo1.maven.org/maven2/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.${packaging.type}
> [info] downloading
> https://repo1.maven.org/maven2/org/apache/kafka/kafka-streams-test-utils/2.1.0/kafka-streams-test-utils-2.1.0.jar
> ...
> [info] [SUCCESSFUL ]
> org.apache.kafka#kafka-streams-test-utils;2.1.0!kafka-streams-test-utils.jar
> (344ms)
> [warn] ::
> [warn] :: FAILED DOWNLOADS ::
> [warn] :: ^ see resolution messages for details ^ ::
> [warn] ::
> [warn] :: javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}
> [warn] ::
> [trace] Stack trace suppressed: run 'last *:ssExtractDependencies' for the
> full output.
> [trace] Stack trace suppressed: run 'last *:update' for the full output.
> [error] (*:ssExtractDependencies) sbt.ResolveException: download failed:
> javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}
> [error] (*:update) sbt.ResolveException: download failed:
> javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}
> [error] Total time: 8 s, completed 16-Dec-2018 19:27:21
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=384M;
> support was removed in 8.0See complete log in href="file:/C:/Users/sacha/.IdeaIC2017.1/system/log/sbt.last.log">file:/C:/Users/sacha/.IdeaIC2017.1/system/log/sbt.last.log
> {code}
> This seems to be a common issue with bad dependency from Kafka to
> javax.ws.rs-api.
> if I drop the Kafka version down to 2.0.0 and add this line to my SBT file
> this error goes away
> {code}
> libraryDependencies += "javax.ws.rs" % "javax.ws.rs-api" % "2.1"
> artifacts(Artifact("javax.ws.rs-api", "jar", "jar"))`
> {code}
>
> However I would like to work with 2.1.0 version.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7741) Bad dependency via SBT
[
https://issues.apache.org/jira/browse/KAFKA-7741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin P. McCabe updated KAFKA-7741:
---
Fix Version/s: (was: 2.1.1)
> Bad dependency via SBT
> --
>
> Key: KAFKA-7741
> URL: https://issues.apache.org/jira/browse/KAFKA-7741
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.0.0, 2.0.1, 2.1.0
> Environment: Windows 10 professional, IntelliJ IDEA 2017.1
>Reporter: sacha barber
>Assignee: John Roesler
>Priority: Major
> Fix For: 2.2.0, 2.0.2
>
>
> I am using the Kafka-Streams-Scala 2.1.0 JAR.
> And if I create a new Scala project using SBT with these dependencies
> {code}
> name := "ScalaKafkaStreamsDemo"
> version := "1.0"
> scalaVersion := "2.12.1"
> libraryDependencies += "org.apache.kafka" %% "kafka" % "2.0.0"
> libraryDependencies += "org.apache.kafka" % "kafka-clients" % "2.0.0"
> libraryDependencies += "org.apache.kafka" % "kafka-streams" % "2.0.0"
> libraryDependencies += "org.apache.kafka" %% "kafka-streams-scala" % "2.0.0"
> //TEST
> libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.5" % Test
> libraryDependencies += "org.apache.kafka" % "kafka-streams-test-utils" %
> "2.0.0" % Test
> {code}
> I get this error
>
> {code}
> SBT 'ScalaKafkaStreamsDemo' project refresh failed
> Error:Error while importing SBT project:...[info] Resolving
> jline#jline;2.14.1 ...
> [warn] [FAILED ]
> javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}: (0ms)
> [warn] local: tried
> [warn]
> C:\Users\sacha\.ivy2\local\javax.ws.rs\javax.ws.rs-api\2.1.1\${packaging.type}s\javax.ws.rs-api.${packaging.type}
> [warn] public: tried
> [warn]
> https://repo1.maven.org/maven2/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.${packaging.type}
> [info] downloading
> https://repo1.maven.org/maven2/org/apache/kafka/kafka-streams-test-utils/2.1.0/kafka-streams-test-utils-2.1.0.jar
> ...
> [info] [SUCCESSFUL ]
> org.apache.kafka#kafka-streams-test-utils;2.1.0!kafka-streams-test-utils.jar
> (344ms)
> [warn] ::
> [warn] :: FAILED DOWNLOADS ::
> [warn] :: ^ see resolution messages for details ^ ::
> [warn] ::
> [warn] :: javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}
> [warn] ::
> [trace] Stack trace suppressed: run 'last *:ssExtractDependencies' for the
> full output.
> [trace] Stack trace suppressed: run 'last *:update' for the full output.
> [error] (*:ssExtractDependencies) sbt.ResolveException: download failed:
> javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}
> [error] (*:update) sbt.ResolveException: download failed:
> javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}
> [error] Total time: 8 s, completed 16-Dec-2018 19:27:21
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=384M;
> support was removed in 8.0See complete log in href="file:/C:/Users/sacha/.IdeaIC2017.1/system/log/sbt.last.log">file:/C:/Users/sacha/.IdeaIC2017.1/system/log/sbt.last.log
> {code}
> This seems to be a common issue with bad dependency from Kafka to
> javax.ws.rs-api.
> if I drop the Kafka version down to 2.0.0 and add this line to my SBT file
> this error goes away
> {code}
> libraryDependencies += "javax.ws.rs" % "javax.ws.rs-api" % "2.1"
> artifacts(Artifact("javax.ws.rs-api", "jar", "jar"))`
> {code}
>
> However I would like to work with 2.1.0 version.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7818) Make the KafkaConsumer AutoCloseable
[ https://issues.apache.org/jira/browse/KAFKA-7818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin P. McCabe updated KAFKA-7818: --- Affects Version/s: (was: 2.1.0) Fix Version/s: (was: 2.1.1) > Make the KafkaConsumer AutoCloseable > > > Key: KAFKA-7818 > URL: https://issues.apache.org/jira/browse/KAFKA-7818 > Project: Kafka > Issue Type: Improvement > Components: clients >Reporter: Fokko Driesprong >Priority: Major > Labels: needs-kip > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7834) Extend collected logs in system test services to include heap dumps
[
https://issues.apache.org/jira/browse/KAFKA-7834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin P. McCabe updated KAFKA-7834:
---
Fix Version/s: (was: 2.1.1)
> Extend collected logs in system test services to include heap dumps
> ---
>
> Key: KAFKA-7834
> URL: https://issues.apache.org/jira/browse/KAFKA-7834
> Project: Kafka
> Issue Type: Improvement
> Components: system tests
>Reporter: Konstantine Karantasis
>Assignee: Konstantine Karantasis
>Priority: Major
> Fix For: 1.0.3, 1.1.2, 3.0.0, 2.2.0, 2.0.2
>
>
> Overall I'd suggest enabling by default:
> {\{-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="}}
> in the major system test services, so that a heap dump is captured on OOM.
> Given these flags, we should also extend the set of collected logs in each
> service to include the predetermined filename for the heap dump.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7856) Cryptographic Issues by Insufficient Entropy
[ https://issues.apache.org/jira/browse/KAFKA-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16751663#comment-16751663 ] ASF GitHub Bot commented on KAFKA-7856: --- cmccabe commented on pull request #6184: KAFKA-7856: Fix vulnerability CWE-331 URL: https://github.com/apache/kafka/pull/6184 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Cryptographic Issues by Insufficient Entropy > > > Key: KAFKA-7856 > URL: https://issues.apache.org/jira/browse/KAFKA-7856 > Project: Kafka > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Victor Sartori >Priority: Major > Labels: patch, pull-request-available, security > Fix For: 2.1.1 > > > We pass the kakfa client in security analisys ans this scans reports: > CWE-331 - Flaw medium,SANS TOP 25 > [https://cwe.mitre.org/data/definitions/331.html] > > A PR on github is present. (https://github.com/apache/kafka/pull/6184) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-7856) Cryptographic Issues by Insufficient Entropy
[ https://issues.apache.org/jira/browse/KAFKA-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin P. McCabe resolved KAFKA-7856. Resolution: Invalid > Cryptographic Issues by Insufficient Entropy > > > Key: KAFKA-7856 > URL: https://issues.apache.org/jira/browse/KAFKA-7856 > Project: Kafka > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Victor Sartori >Priority: Major > Labels: patch, pull-request-available, security > Fix For: 2.1.1 > > > We pass the kakfa client in security analisys ans this scans reports: > CWE-331 - Flaw medium,SANS TOP 25 > [https://cwe.mitre.org/data/definitions/331.html] > > A PR on github is present. (https://github.com/apache/kafka/pull/6184) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7856) Cryptographic Issues by Insufficient Entropy
[ https://issues.apache.org/jira/browse/KAFKA-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16751662#comment-16751662 ] Colin P. McCabe commented on KAFKA-7856: I don't see any cryptographic issue here. The random number generator is being used for injecting some randomness into the retry delay. Even if an attacker could predict exactly when retries would happen, there is no security implication that I can see. If you're a bad guy, then just write your own client that retries as quickly as possible. > Cryptographic Issues by Insufficient Entropy > > > Key: KAFKA-7856 > URL: https://issues.apache.org/jira/browse/KAFKA-7856 > Project: Kafka > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Victor Sartori >Priority: Major > Labels: patch, pull-request-available, security > Fix For: 2.1.1 > > > We pass the kakfa client in security analisys ans this scans reports: > CWE-331 - Flaw medium,SANS TOP 25 > [https://cwe.mitre.org/data/definitions/331.html] > > A PR on github is present. (https://github.com/apache/kafka/pull/6184) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-7273) Converters should have access to headers.
[
https://issues.apache.org/jira/browse/KAFKA-7273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeremy Custenborder reassigned KAFKA-7273:
--
Assignee: Jeremy Custenborder
> Converters should have access to headers.
> -
>
> Key: KAFKA-7273
> URL: https://issues.apache.org/jira/browse/KAFKA-7273
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Reporter: Jeremy Custenborder
>Assignee: Jeremy Custenborder
>Priority: Major
>
> I found myself wanting to build a converter that stored additional type
> information within headers. The converter interface does not allow a
> developer to access to the headers in a Converter. I'm not suggesting that we
> change the method for serializing them, rather that
> *org.apache.kafka.connect.header.Headers* be passed in for *fromConnectData*
> and *toConnectData*. For example something like this.
> {code:java}
> import org.apache.kafka.connect.data.Schema;
> import org.apache.kafka.connect.data.SchemaAndValue;
> import org.apache.kafka.connect.header.Headers;
> import org.apache.kafka.connect.storage.Converter;
> public interface Converter {
> default byte[] fromConnectData(String topic, Headers headers, Schema
> schema, Object object) {
> return fromConnectData(topic, schema, object);
> }
> default SchemaAndValue toConnectData(String topic, Headers headers, byte[]
> payload) {
> return toConnectData(topic, payload);
> }
> void configure(Map var1, boolean var2);
> byte[] fromConnectData(String var1, Schema var2, Object var3);
> SchemaAndValue toConnectData(String var1, byte[] var2);
> }
> {code}
> This would be a similar approach to what was already done with
> ExtendedDeserializer and ExtendedSerializer in the Kafka client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KAFKA-7796) structured streaming fetched wrong current offset from kafka
[
https://issues.apache.org/jira/browse/KAFKA-7796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryne Yang resolved KAFKA-7796.
--
Resolution: Invalid
Not related to kafka, it's actually spark structured streaming integer
overflow: https://issues.apache.org/jira/browse/SPARK-26718
> structured streaming fetched wrong current offset from kafka
>
>
> Key: KAFKA-7796
> URL: https://issues.apache.org/jira/browse/KAFKA-7796
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Environment: Linux, Centos 7
>Reporter: Ryne Yang
>Priority: Major
>
> when running spark structured streaming using lib: `"org.apache.spark" %%
> "spark-sql-kafka-0-10" % "2.4.0"`, we keep getting error regarding current
> offset fetching:
> {code:java}
> Caused by: org.apache.spark.SparkException: Job aborted due to stage failure:
> Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in
> stage 0.0 (TID 3, qa2-hdp-4.acuityads.org, executor 2):
> java.lang.AssertionError: assertion failed: latest offs
> et -9223372036854775808 does not equal -1
> at scala.Predef$.assert(Predef.scala:170)
> at
> org.apache.spark.sql.kafka010.KafkaMicroBatchInputPartitionReader.resolveRange(KafkaMicroBatchReader.scala:371)
> at
> org.apache.spark.sql.kafka010.KafkaMicroBatchInputPartitionReader.(KafkaMicroBatchReader.scala:329)
> at
> org.apache.spark.sql.kafka010.KafkaMicroBatchInputPartition.createPartitionReader(KafkaMicroBatchReader.scala:314)
> at
> org.apache.spark.sql.execution.datasources.v2.DataSourceRDD.compute(DataSourceRDD.scala:42)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
> at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
> at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
> at org.apache.spark.scheduler.Task.run(Task.scala:121)
> at
> org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> for some reason, looks like fetchLatestOffset returned a Long.MIN_VALUE for
> one of the partitions. I checked the structured streaming checkpoint, that
> was correct, it's the currentAvailableOffset was set to Long.MIN_VALUE.
> kafka broker version: 1.1.0.
> lib we used:
> {{libraryDependencies += "org.apache.spark" %% "spark-sql-kafka-0-10" %
> "2.4.0" }}
> how to reproduce:
> basically we started a structured streamer and subscribed a topic of 4
> partitions. then produced some messages into topic, job crashed and logged
> the stacktrace like above.
> also the committed offsets seem fine as we see in the logs:
> {code:java}
> === Streaming Query ===
> Identifier: [id = c46c67ee-3514-4788-8370-a696837b21b1, runId =
> 31878627-d473-4ee8-955d-d4d3f3f45eb9]
> Current Committed Offsets: {KafkaV2[Subscribe[REVENUEEVENT]]:
> {"REVENUEEVENT":{"0":1}}}
> Current Available Offsets: {KafkaV2[Subscribe[REVENUEEVENT]]:
> {"REVENUEEVENT":{"0":-9223372036854775808}}}
> {code}
> so spark streaming recorded the correct value for partition: 0, but the
> current available offsets returned from kafka is showing Long.MIN_VALUE.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (KAFKA-7796) structured streaming fetched wrong current offset from kafka
[
https://issues.apache.org/jira/browse/KAFKA-7796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryne Yang closed KAFKA-7796.
> structured streaming fetched wrong current offset from kafka
>
>
> Key: KAFKA-7796
> URL: https://issues.apache.org/jira/browse/KAFKA-7796
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Environment: Linux, Centos 7
>Reporter: Ryne Yang
>Priority: Major
>
> when running spark structured streaming using lib: `"org.apache.spark" %%
> "spark-sql-kafka-0-10" % "2.4.0"`, we keep getting error regarding current
> offset fetching:
> {code:java}
> Caused by: org.apache.spark.SparkException: Job aborted due to stage failure:
> Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in
> stage 0.0 (TID 3, qa2-hdp-4.acuityads.org, executor 2):
> java.lang.AssertionError: assertion failed: latest offs
> et -9223372036854775808 does not equal -1
> at scala.Predef$.assert(Predef.scala:170)
> at
> org.apache.spark.sql.kafka010.KafkaMicroBatchInputPartitionReader.resolveRange(KafkaMicroBatchReader.scala:371)
> at
> org.apache.spark.sql.kafka010.KafkaMicroBatchInputPartitionReader.(KafkaMicroBatchReader.scala:329)
> at
> org.apache.spark.sql.kafka010.KafkaMicroBatchInputPartition.createPartitionReader(KafkaMicroBatchReader.scala:314)
> at
> org.apache.spark.sql.execution.datasources.v2.DataSourceRDD.compute(DataSourceRDD.scala:42)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
> at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
> at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
> at org.apache.spark.scheduler.Task.run(Task.scala:121)
> at
> org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> for some reason, looks like fetchLatestOffset returned a Long.MIN_VALUE for
> one of the partitions. I checked the structured streaming checkpoint, that
> was correct, it's the currentAvailableOffset was set to Long.MIN_VALUE.
> kafka broker version: 1.1.0.
> lib we used:
> {{libraryDependencies += "org.apache.spark" %% "spark-sql-kafka-0-10" %
> "2.4.0" }}
> how to reproduce:
> basically we started a structured streamer and subscribed a topic of 4
> partitions. then produced some messages into topic, job crashed and logged
> the stacktrace like above.
> also the committed offsets seem fine as we see in the logs:
> {code:java}
> === Streaming Query ===
> Identifier: [id = c46c67ee-3514-4788-8370-a696837b21b1, runId =
> 31878627-d473-4ee8-955d-d4d3f3f45eb9]
> Current Committed Offsets: {KafkaV2[Subscribe[REVENUEEVENT]]:
> {"REVENUEEVENT":{"0":1}}}
> Current Available Offsets: {KafkaV2[Subscribe[REVENUEEVENT]]:
> {"REVENUEEVENT":{"0":-9223372036854775808}}}
> {code}
> so spark streaming recorded the correct value for partition: 0, but the
> current available offsets returned from kafka is showing Long.MIN_VALUE.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-7869) Refactor RocksDBConfigSetter API to separate DBOptions and CFOptions
Guozhang Wang created KAFKA-7869:
Summary: Refactor RocksDBConfigSetter API to separate DBOptions
and CFOptions
Key: KAFKA-7869
URL: https://issues.apache.org/jira/browse/KAFKA-7869
Project: Kafka
Issue Type: Improvement
Components: streams
Reporter: Guozhang Wang
Current RocksDBConfigSetter has the following API:
{code}
void setConfig(final String storeName, final Options options, final Map configs);
{code}
Where `Options` contains configurations for both db-level and cf-level of
RocksDB.
As we move on to have multiple CFs following KIP-258, it's better to refactor
it into
{code}
void setConfig(final String storeName, final DBOptions dbOptions, final
ColumnFamilyOptions cfOptions, final Map configs);
{code}
And then inside the internal implementation, if only the default CF is used, we
can still use the other constructor of `Options` that takes both a DBOptions
and CFOptions object as parameters.
This should be started only after KIP-258 is finished.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KAFKA-7855) Kafka Streams Maven Archetype quickstart fails to compile out of the box
[ https://issues.apache.org/jira/browse/KAFKA-7855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax reassigned KAFKA-7855: -- Assignee: Kristian Aurlien > Kafka Streams Maven Archetype quickstart fails to compile out of the box > > > Key: KAFKA-7855 > URL: https://issues.apache.org/jira/browse/KAFKA-7855 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.1.0 > Environment: Java 8, OS X 10.13.6 >Reporter: Michael Drogalis >Assignee: Kristian Aurlien >Priority: Major > Labels: newbie++ > Attachments: output.log > > > When I follow the [quickstart > tutorial|https://kafka.apache.org/21/documentation/streams/tutorial] and > issue the command to set up a new Maven project, the generated example fails > to compile. Adding a Produced.with() on the source seems to fix this. I've > attached the compiler output. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7855) Kafka Streams Maven Archetype quickstart fails to compile out of the box
[ https://issues.apache.org/jira/browse/KAFKA-7855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16751391#comment-16751391 ] Matthias J. Sax commented on KAFKA-7855: Thanks for the PR. Added you to the list of contributors and assigned the ticket to you. You can now also self-assign tickets. > Kafka Streams Maven Archetype quickstart fails to compile out of the box > > > Key: KAFKA-7855 > URL: https://issues.apache.org/jira/browse/KAFKA-7855 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.1.0 > Environment: Java 8, OS X 10.13.6 >Reporter: Michael Drogalis >Assignee: Kristian Aurlien >Priority: Major > Labels: newbie++ > Attachments: output.log > > > When I follow the [quickstart > tutorial|https://kafka.apache.org/21/documentation/streams/tutorial] and > issue the command to set up a new Maven project, the generated example fails > to compile. Adding a Produced.with() on the source seems to fix this. I've > attached the compiler output. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7782) cleanup policy for offsets topic should be configurable
[ https://issues.apache.org/jira/browse/KAFKA-7782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16751343#comment-16751343 ] Nir Barel commented on KAFKA-7782: -- Hi, currently compact policy is not working on windows and therefore we use delete policy can you elaborate more why running with delete policy isn’t good? > cleanup policy for offsets topic should be configurable > --- > > Key: KAFKA-7782 > URL: https://issues.apache.org/jira/browse/KAFKA-7782 > Project: Kafka > Issue Type: Improvement > Components: core >Affects Versions: 2.1.0 >Reporter: Nir Barel >Priority: Major > Labels: windows > > Hi, > For offsets topic there is no option to change the log cleanup policy from > compact to delete because it is set hard-coded in code here: > [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/coordinator/group/GroupCoordinator.scala#L68] > The only way to override it is to alter the topic at runtime which required > to check the settings and set it after every restart of Kafka. > I am suggesting to configure it via properties with a default value of > "compact" > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7793) Improve the Trogdor command-line
[ https://issues.apache.org/jira/browse/KAFKA-7793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16751347#comment-16751347 ] ASF GitHub Bot commented on KAFKA-7793: --- cmccabe commented on pull request #6133: KAFKA-7793: Improve the Trogdor command line. URL: https://github.com/apache/kafka/pull/6133 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Improve the Trogdor command-line > > > Key: KAFKA-7793 > URL: https://issues.apache.org/jira/browse/KAFKA-7793 > Project: Kafka > Issue Type: Improvement >Reporter: Colin P. McCabe >Priority: Major > > Improve the Trogdor command-line. It should be easier to launch tasks from a > task spec in a file. It should be easier to list the currently-running tasks > in a readable way. We should be able to filter the currently-running tasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7782) cleanup policy for offsets topic should be configurable
[
https://issues.apache.org/jira/browse/KAFKA-7782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16751289#comment-16751289
]
Lee Dongjin commented on KAFKA-7782:
{quote}It's is by design that the cleanup policy should not be changed.{quote}
Then, the offsets topic can be altered with kafka-configs.sh *is* the bug.
Right?
> cleanup policy for offsets topic should be configurable
> ---
>
> Key: KAFKA-7782
> URL: https://issues.apache.org/jira/browse/KAFKA-7782
> Project: Kafka
> Issue Type: Improvement
> Components: core
>Affects Versions: 2.1.0
>Reporter: Nir Barel
>Priority: Major
> Labels: windows
>
> Hi,
> For offsets topic there is no option to change the log cleanup policy from
> compact to delete because it is set hard-coded in code here:
> [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/coordinator/group/GroupCoordinator.scala#L68]
> The only way to override it is to alter the topic at runtime which required
> to check the settings and set it after every restart of Kafka.
> I am suggesting to configure it via properties with a default value of
> "compact"
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-7868) Move __consumer_offsets schema to the clients
Charly Molter created KAFKA-7868: Summary: Move __consumer_offsets schema to the clients Key: KAFKA-7868 URL: https://issues.apache.org/jira/browse/KAFKA-7868 Project: Kafka Issue Type: Improvement Components: clients Reporter: Charly Molter Currently the schema for data stored in __consumer_offsets is declared in kafka.coordinator.group.GroupMetadataManager. However, it is common for users to consumer __consumer_offsets to monitor the backlog of consumers. Therefore it would make sense to move this schema to org.apache.kafka.common this would avoid these users to have to redefine the schema somewhere else. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7855) Kafka Streams Maven Archetype quickstart fails to compile out of the box
[ https://issues.apache.org/jira/browse/KAFKA-7855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16751191#comment-16751191 ] ASF GitHub Bot commented on KAFKA-7855: --- aurlien commented on pull request #6194: KAFKA-7855: [WIP] Kafka Streams Maven Archetype quickstart fails to compile out of the box URL: https://github.com/apache/kafka/pull/6194 In the LineSplit.java example, the untyped KStream is resolved to `KStream`. `value.split(..)` then fails to build, because value is of type Object. Other possible solutions would be to add the type to the builder: `builder.stream(...).flatmap(..).to(...)`, or `builder.stream(..., Consumed.with(Serdes.String(), Serdes.String()))`. This change matches the code in the [tutorial](http://kafka.apache.org/21/documentation/streams/tutorial). WIP: It should be tested during compile that the Java resource files compiles. However, I haven't found any way to achieve this. Feedback appreciated! ### Committer Checklist (excluded from commit message) - [ x ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Kafka Streams Maven Archetype quickstart fails to compile out of the box > > > Key: KAFKA-7855 > URL: https://issues.apache.org/jira/browse/KAFKA-7855 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.1.0 > Environment: Java 8, OS X 10.13.6 >Reporter: Michael Drogalis >Priority: Major > Labels: newbie++ > Attachments: output.log > > > When I follow the [quickstart > tutorial|https://kafka.apache.org/21/documentation/streams/tutorial] and > issue the command to set up a new Maven project, the generated example fails > to compile. Adding a Produced.with() on the source seems to fix this. I've > attached the compiler output. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KAFKA-7223) KIP-328: Add in-memory Suppression
[ https://issues.apache.org/jira/browse/KAFKA-7223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler closed KAFKA-7223. --- > KIP-328: Add in-memory Suppression > -- > > Key: KAFKA-7223 > URL: https://issues.apache.org/jira/browse/KAFKA-7223 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: John Roesler >Assignee: John Roesler >Priority: Major > Fix For: 2.1.0 > > > As described in > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-328%3A+Ability+to+suppress+updates+for+KTables.] > > This ticket is to implement Suppress, but only for in-memory buffers. > (depends on KAFKA-7222) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-7223) KIP-328: Add in-memory Suppression
[ https://issues.apache.org/jira/browse/KAFKA-7223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler resolved KAFKA-7223. - Resolution: Fixed > KIP-328: Add in-memory Suppression > -- > > Key: KAFKA-7223 > URL: https://issues.apache.org/jira/browse/KAFKA-7223 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: John Roesler >Assignee: John Roesler >Priority: Major > Fix For: 2.1.0 > > > As described in > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-328%3A+Ability+to+suppress+updates+for+KTables.] > > This ticket is to implement Suppress, but only for in-memory buffers. > (depends on KAFKA-7222) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7867) Broker fails after corrupted page table
Maurits Hartman created KAFKA-7867:
--
Summary: Broker fails after corrupted page table
Key: KAFKA-7867
URL: https://issues.apache.org/jira/browse/KAFKA-7867
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 2.0.0
Environment: openjdk version "11.0.1" 2018-10-16
OpenJDK Runtime Environment (build 11.0.1+13-Ubuntu-2ubuntu1)
OpenJDK 64-Bit Server VM (build 11.0.1+13-Ubuntu-2ubuntu1, mixed mode, sharing)
Ubuntu 18.10 (Cosmic)
Linux kernel 4.18.0-13
4x Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
8GB RAM
Reporter: Maurits Hartman
We noticed one of the brokers in our cluster was down. It was no longer leader
for its partitions, nor was it following the other broker's partitions. Kafka
was still running apparently though (the log was still being appended to).
In the systemd journal we noticed a "corrupted page table" error that happened
a couple of hours before:
{code:java}
Jan 24 03:22:07 kafka9 kernel: kafka-request-h: Corrupted page table at address
8402805f8 Jan 24 03:22:07 kafka9 kernel: Bad pagetable: 000d [#1] SMP PTI Jan
24 03:22:07 kafka9 kernel: CPU: 3 PID: 3025 Comm: kafka-request-h Not tainted
4.18.0-13-generic #14-Ubuntu Jan 24 03:22:07 kafka9 kernel: Hardware name:
DigitalOcean Droplet, BIOS 20171212 12/12/2017 Jan 24 03:22:07 kafka9 kernel:
RIP: 0033:0x7f8878c63f4f Jan 24 03:22:07 kafka9 kernel: Code: 00 00 0f 83 fa 05
00 00 4d 89 9f 18 01 00 00 41 0f 0d 8b 00 01 00 00 bb a8 00 05 08 49 bb 00 00
00 00 08 00 00 00 4d 8d 1c db <4d> 8b 9b b8 00 00 00 4c 89 18 c7 40 08 a8 00 05
08 c7 40 0c 00 00 Jan 24 03:22:07 kafka9 kernel: RSP: 002b:7f7fee2924b0
EFLAGS: 00010283 Jan 24 03:22:07 kafka9 kernel: RAX: f757ba48 RBX:
080500a8 RCX: f757ba28 Jan 24 03:22:07 kafka9 kernel: RDX:
f757b9b8 RSI: f757b9ec RDI: 005596e0 Jan 24 03:22:07
kafka9 kernel: RBP: R08: d1d961e0 R09:
00559600 Jan 24 03:22:07 kafka9 kernel: R10: 00559600 R11:
000840280540 R12: Jan 24 03:22:07 kafka9 kernel: R13:
cea8da78 R14: 0001 R15: 7fac3800 Jan 24 03:22:07
kafka9 kernel: FS: 7f7fee293700 GS: Jan 24 03:22:07 kafka9
kernel: Modules linked in: isofs nls_iso8859_1 kvm_intel kvm irqbypass
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel joydev input_leds serio_raw
ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 xt_hl ip6t_rt nf_conntrack_ipv6
nf_defrag_ipv6 ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG
xt_limit xt_tcpudp xt_ad Jan 24 03:22:07 kafka9 kernel: crypto_simd cryptd
glue_helper psmouse virtio_blk floppy virtio_net net_failover virtio_scsi
failover Jan 24 03:22:07 kafka9 kernel: ---[ end trace 23d8fa22733ef791 ]---
Jan 24 03:22:07 kafka9 kernel: RIP: 0033:0x7f8878c63f4f Jan 24 03:22:07 kafka9
kernel: Code: 00 00 0f 83 fa 05 00 00 4d 89 9f 18 01 00 00 41 0f 0d 8b 00 01 00
00 bb a8 00 05 08 49 bb 00 00 00 00 08 00 00 00 4d 8d 1c db <4d> 8b 9b b8 00 00
00 4c 89 18 c7 40 08 a8 00 05 08 c7 40 0c 00 00 Jan 24 03:22:07 kafka9 kernel:
RSP: 002b:7f7fee2924b0 EFLAGS: 00010283 Jan 24 03:22:07 kafka9 kernel: RAX:
f757ba48 RBX: 080500a8 RCX: f757ba28 Jan 24 03:22:07
kafka9 kernel: RDX: f757b9b8 RSI: f757b9ec RDI:
005596e0 Jan 24 03:22:07 kafka9 kernel: RBP: R08:
d1d961e0 R09: 00559600 Jan 24 03:22:07 kafka9 kernel: R10:
00559600 R11: 000840280540 R12: Jan 24 03:22:07
kafka9 kernel: R13: cea8da78 R14: 0001 R15:
7fac3800 Jan 24 03:22:07 kafka9 kernel: FS: 7f7fee293700()
GS:97a29fd8() knlGS: Jan 24 03:22:07 kafka9 kernel:
CS: 0010 DS: ES: CR0: 80050033 Jan 24 03:22:07 kafka9 kernel:
CR2: 0008402805f8 CR3: 0001f9f32006 CR4: 007606e0 Jan 24
03:22:07 kafka9 kernel: DR0: DR1: DR2:
Jan 24 03:22:07 kafka9 kernel: DR3: DR6:
fffe0ff0 DR7: 0400 Jan 24 03:22:07 kafka9 kernel: PKRU:
5554{code}
Broker configuration:
{code:java}
broker.id=9
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/mnt/volume_kafka9a/kafka,/mnt/volume_kafka9b/kafka,/mnt/volume_kafka9c/kafka
num.partitions=1
num.recovery.threads.per.data.dir=4
offsets.topic.replication.factor=2
transaction.state.log.replication.factor=2
transaction.state.log.min.isr=2
auto.leader.rebalance.enable=true
leader.imbalance.check.interval.seconds=30
unclean.leader.election.enable=true
log.retention.hours=168
log.segment.bytes=8388608
log.retention.check.interval.ms=30
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
defa
[jira] [Commented] (KAFKA-7854) Behavior change in controller picking up partition reassignment tasks since 1.1.0
[
https://issues.apache.org/jira/browse/KAFKA-7854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16750875#comment-16750875
]
Tom Bentley commented on KAFKA-7854:
Just for context, my original thoughts about the API for partition reassignment
were in KIP-179, but I view it as being a bit too basic for the sorts of things
I'd like to be able to do. It was enough to replace the direct zookeeper access
required by {{kafka-reassign-partitions.sh}}, but it didn't tackle some of the
other annoying things about partition reassignment, like the fact that you
can't start new reassignments when some are already in progress. For that
reason I thought it better to work on a more ambitious set of APIs, and KIPs
236 and 240 were the result (and why I withdrew 179). What I didn't publish
were my ideas for the actual reassignment API. I still have those details
buried away somewhere, but I have lacked the time to work on any of this stuff
properly over the last year. I guess I could try to write up my thoughts about
that API if people would be interested.
> Behavior change in controller picking up partition reassignment tasks since
> 1.1.0
> -
>
> Key: KAFKA-7854
> URL: https://issues.apache.org/jira/browse/KAFKA-7854
> Project: Kafka
> Issue Type: Improvement
> Components: controller
>Reporter: Zhanxiang (Patrick) Huang
>Priority: Major
>
> After [https://github.com/apache/kafka/pull/4143,] the controller does not
> subscribe to data change on /admin/reassign_partitions any more (in order to
> avoid unnecessarily loading the reassignment data again after controller
> updating the znode) as opposed to the previous kafka versions. However, there
> are systems built around kafka relying on the previous behavior to
> incrementally update the list of partition reassignment since kafka does not
> natively support that.
>
> For example, [cruise control|https://github.com/linkedin/cruise-control] can
> rely on the previous behavior (controller listening to data changes) to
> maintain the reassignment concurrency by dynamically updating the data in the
> reassignment znode instead of waiting for the current batch to finish and
> doing reassignment batch by batch, which can significantly reduce the
> rebalance time in production clusters. Although directly updating the znode
> can somehow be viewed as an anti-pattern in the long term, this is necessary
> since kafka does not natively support incrementally submit more reassignment
> tasks. However, after our kafka clusters migrate from 0.11 to 2.0, cruise
> control no longer works because the controller behavior has changed. This
> reveals the following problems:
> * These behavior changes may be viewed as internal changes so compatibility
> is not guaranteed but I think by convention people do view this as public
> interfaces and rely on the compatibility. In this case, I think we should
> clearly document the data contract for the partition reassignment task to
> avoid misusage and making controller changes that break the defined data
> contract. There may be other cases (e.g. topic deletion) whose data contracts
> need to be clearly defined and we should keep it in mind when making
> controller changes.
> * Kafka does not natively support incrementally submit more reassignment
> tasks. If we do want to support that nicely, we should consider change how we
> store the reassignment data to store the data in child nodes and let the
> controller listen on child node changes, similar to what we do for
> /admin/delete_topics.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
