Re: Splitting partition may cause message loss for consumers
Thanks for your quick response. Yeah, agree with that. (also replied on the issue) 2021年2月19日(金) 16:25 Luke Chen : > Hi Okada san, > Yes, I agree the "latest" setting in this situation is not good, and we > should document it. > But I don't think we should change the default auto.offset.reset setting to > the earliest. > The auto.offset.reset setting starts before kafka V1.0, which means, there > are already a lot of users using it and get used to it now. > > Thanks. > Luke > > On Fri, Feb 19, 2021 at 2:24 PM Haruki Okada wrote: > > > Hi, Kafka. > > > > Recently I noticed that splitting partition may cause message delivery > loss > > for consumers with auto.offset.reset=latest. > > > > I described details in https://issues.apache.org/jira/browse/KAFKA-12261 > . > > > > Since delivery loss is undesired in most cases, I think this should be > > described in ConsumerConfig.AUTO_OFFSET_RESET_DOC (or somewhere) at > least. > > > > What do you think? > > > > > > Thanks, > > -- > > > > Okada Haruki > > ocadar...@gmail.com > > > > > -- Okada Haruki ocadar...@gmail.com
Re: Splitting partition may cause message loss for consumers
Hi Okada san, Yes, I agree the "latest" setting in this situation is not good, and we should document it. But I don't think we should change the default auto.offset.reset setting to the earliest. The auto.offset.reset setting starts before kafka V1.0, which means, there are already a lot of users using it and get used to it now. Thanks. Luke On Fri, Feb 19, 2021 at 2:24 PM Haruki Okada wrote: > Hi, Kafka. > > Recently I noticed that splitting partition may cause message delivery loss > for consumers with auto.offset.reset=latest. > > I described details in https://issues.apache.org/jira/browse/KAFKA-12261 . > > Since delivery loss is undesired in most cases, I think this should be > described in ConsumerConfig.AUTO_OFFSET_RESET_DOC (or somewhere) at least. > > What do you think? > > > Thanks, > -- > > Okada Haruki > ocadar...@gmail.com > >
[jira] [Created] (KAFKA-12346) punctuate is called at twice the duration passed as the first argument to Processor.Schedule (with PunctuationType.WALL_CLOCK_TIME)
Arindam Ray created KAFKA-12346:
---
Summary: punctuate is called at twice the duration passed as the
first argument to Processor.Schedule (with PunctuationType.WALL_CLOCK_TIME)
Key: KAFKA-12346
URL: https://issues.apache.org/jira/browse/KAFKA-12346
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 2.7.0
Reporter: Arindam Ray
A stream transform called with the idiom below causes punctuate to be called at
twice the duration of the argument passed
{code:java}
.transform(new TransformerSupplier[String, TimeStampedString,
KeyValue[String, TimeStampedString]]() {
override def get(): Transformer[String, TimeStampedString,
KeyValue[String, TimeStampedString]] = new Transformer[String,
TimeStampedString, KeyValue[String, TimeStampedString]] {override
def init(context: ProcessorContext): Unit = {
val store =
context.getStateStore(stateStoreName).asInstanceOf[KeyValueStore[String,
ValueAndTimestamp[TimeStampedString]]]
context.schedule(scanFrequency,
PunctuationType.WALL_CLOCK_TIME,
new Punctuator {
override def punctuate(timestamp: Long): Unit = {
logger.info(s"Punctuate invoked with timestamp :
${Instant.ofEpochMilli(timestamp)}")
}
}
)
}override def transform(key: String, value:
TimeStampedString): KeyValue[String, TimeStampedString] = {
// no need to return anything here, the Punctuator will emit the
records when necessary
null
}override def close(): Unit = {}
}
}, /**
* register that this Transformer needs to be connected to our state
store.
*/
stateStoreName
)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Splitting partition may cause message loss for consumers
Hi, Kafka. Recently I noticed that splitting partition may cause message delivery loss for consumers with auto.offset.reset=latest. I described details in https://issues.apache.org/jira/browse/KAFKA-12261 . Since delivery loss is undesired in most cases, I think this should be described in ConsumerConfig.AUTO_OFFSET_RESET_DOC (or somewhere) at least. What do you think? Thanks, -- Okada Haruki ocadar...@gmail.com
[jira] [Resolved] (KAFKA-12258) Change the BatchAccumulator to split records into batches
[
https://issues.apache.org/jira/browse/KAFKA-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Gustafson resolved KAFKA-12258.
-
Resolution: Fixed
> Change the BatchAccumulator to split records into batches
> -
>
> Key: KAFKA-12258
> URL: https://issues.apache.org/jira/browse/KAFKA-12258
> Project: Kafka
> Issue Type: Sub-task
> Components: controller, core
>Affects Versions: 2.8.0
>Reporter: Alok Nikhil
>Assignee: Jose Armando Garcia Sancio
>Priority: Major
> Labels: kip-500
>
> Modify the `BatchAccumulator.append contract` to support splitting a batch of
> records whose size is greater than the maximum allowed size (of 1048576
> currently) into batches to avoid RaftClient failures such as
> {code:java}
> leader=0, leaderEpoch=0, partitionEpoch=0) at version 0),
> ApiMessageAndVersion(TopicRecord(name='topic-BEHRW',
> topicId=6cRudOGO3yqlsu48RwyPSw) at version 0),
> ApiMessageAndVersion(PartitionRecord(partitionId=0,
> topicId=6cRudOGO3yqlsu48RwyPSw, replicas=[1, 2, 0], isr=[1, 2, 0],
> removingReplicas=null, addingReplicas=null, leader=1, leaderEpoch=0,
> partitionEpoch=0) at version 0)] is 1088890, which exceeds the maximum
> allowed batch size of 1048576 Jan 30 00:13:40 ip-10-0-0-254
> kafka-server-start.sh[633637]: at
> org.apache.kafka.raft.internals.BatchAccumulator.append(BatchAccumulator.java:110)
> Jan 30 00:13:40 ip-10-0-0-254 kafka-server-start.sh[633637]: at
> org.apache.kafka.raft.KafkaRaftClient.scheduleAppend(KafkaRaftClient.java:1885)
> Jan 30 00:13:40 ip-10-0-0-254 kafka-server-start.sh[633637]: at
> org.apache.kafka.raft.metadata.MetaLogRaftShim.scheduleWrite(MetaLogRaftShim.java:60)
> Jan 30 00:13:40 ip-10-0-0-254 kafka-server-start.sh[633637]: at
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:406)
> Jan 30 00:13:40 ip-10-0-0-254 kafka-server-start.sh[633637]: at
> org.apache.kafka.common.utils.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:117)
> Jan 30 00:13:40 ip-10-0-0-254 kafka-server-start.sh[633637]: at
> org.apache.kafka.common.utils.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:192)
> Jan 30 00:13:40 ip-10-0-0-254 kafka-server-start.sh[633637]: at
> org.apache.kafka.common.utils.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:165)
> Jan 30 00:13:40 ip-10-0-0-254 kafka-server-start.sh[633637]: at
> java.base/java.lang.Thread.run(Thread.java:834) Jan 30 00:13:40 ip-10-0-0-254
> kafka-server-start.sh[633637]: [2021-01-30 00:13:40,277] INFO [Controller
> 3000] Reverting to snapshot 2232 (org.apache.kafka.timeline.SnapshotRegistry)
> {code}
> *Example use-case*: Creating 10,000 topics in a single API call
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-9524) Default window retention does not consider grace period
[
https://issues.apache.org/jira/browse/KAFKA-9524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax resolved KAFKA-9524.
Fix Version/s: 2.8.0
Resolution: Fixed
> Default window retention does not consider grace period
> ---
>
> Key: KAFKA-9524
> URL: https://issues.apache.org/jira/browse/KAFKA-9524
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 2.4.0
>Reporter: Michael Bingham
>Assignee: Marco Lotz
>Priority: Minor
> Fix For: 2.8.0
>
>
> In a windowed aggregation, if you specify a window size larger than the
> default window retention (1 day), Streams will implicitly set retention
> accordingly to accommodate windows of that size. For example,
> {code:java}
> .windowedBy(TimeWindows.of(Duration.ofDays(20)))
> {code}
> In this case, Streams will implicitly set window retention to 20 days, and no
> exceptions will occur.
> However, if you also include a non-zero grace period on the window, such as:
> {code:java}
> .windowedBy(TimeWindows.of(Duration.ofDays(20)).grace(Duration.ofMinutes(5)))
> {code}
> In this case, Streams will still implicitly set the window retention 20 days
> (not 20 days + 5 minutes grace), and an exception will be thrown:
> {code:java}
> Exception in thread "main" java.lang.IllegalArgumentException: The retention
> period of the window store KSTREAM-KEY-SELECT-02 must be no smaller
> than its window size plus the grace period. Got size=[172800],
> grace=[30], retention=[172800]{code}
> Ideally, Streams should include grace period when implicitly setting window
> retention.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: About Kafka 2.7.0 source code compilation error
Yes, and also, you can check the readme in kafka github repo for more details. ref: https://github.com/apache/kafka/blob/trunk/README.md Thanks. Luke On Fri, Feb 19, 2021 at 9:51 AM deng ziming wrote: > Hello, please use gradlew, for example `./gradlew jar` `./gradlew idea`, > or you can use the gradle plugin of IDEA. > The `@ nowarn` warn seems to be related to different version of scala and > jdk which you can just ignore. > > > On Feb 18, 2021, at 16:38, 韩可 han...@cvicse.com>> wrote: > > > > Hello! > > > > Recently, we need to build a compilation and development environment for > Kafka 2.7.0 source code. Now the source code can run successfully. However, > when we execute "gradle install" or "gradle build", we will report an > error: @ nowarn annotation does not suppress any warnings. The details are > as follows: > > > > > > > > If you delete the @ nowarn annotation according to the location of the > error, you will still report other errors, so please find out if there is a > good solution, and hope to get your reply. Thank you! > > > > The gradle version used is 6.6 > > The scala version used is 2.13.4 > > > >
Re: About Kafka 2.7.0 source code compilation error
Hello, please use gradlew, for example `./gradlew jar` `./gradlew idea`, or you can use the gradle plugin of IDEA. The `@ nowarn` warn seems to be related to different version of scala and jdk which you can just ignore. > On Feb 18, 2021, at 16:38, 韩可 mailto:han...@cvicse.com>> > wrote: > > Hello! > > Recently, we need to build a compilation and development environment for > Kafka 2.7.0 source code. Now the source code can run successfully. However, > when we execute "gradle install" or "gradle build", we will report an error: > @ nowarn annotation does not suppress any warnings. The details are as > follows: > > > > If you delete the @ nowarn annotation according to the location of the error, > you will still report other errors, so please find out if there is a good > solution, and hope to get your reply. Thank you! > > The gradle version used is 6.6 > The scala version used is 2.13.4 >
[jira] [Created] (KAFKA-12345) KIP-500: AlterIsrManager crashes on broker idle-state
Alok Nikhil created KAFKA-12345: --- Summary: KIP-500: AlterIsrManager crashes on broker idle-state Key: KAFKA-12345 URL: https://issues.apache.org/jira/browse/KAFKA-12345 Project: Kafka Issue Type: Task Components: core Reporter: Alok Nikhil Occasionally, a scheduler thread on a broker crashes with this stack [2021-02-19 01:04:24,683] ERROR Uncaught exception in scheduled task 'send-alter-isr' (kafka.utils.KafkaScheduler) java.lang.NullPointerException at kafka.server.AlterIsrManagerImpl.sendRequest(AlterIsrManager.scala:117) at kafka.server.AlterIsrManagerImpl.propagateIsrChanges(AlterIsrManager.scala:85) at kafka.server.AlterIsrManagerImpl.$anonfun$start$1(AlterIsrManager.scala:66) at kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:114) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) After that the broker is unable to fetch any records from any other broker (and vice versa) [2021-02-19 01:05:07,000] INFO [ReplicaFetcher replicaId=0, leaderId=4, fetcherId=0] Error sending fetch request (sessionId=164432409 2, epoch=957) to node 4: (org.apache.kafka.clients.FetchSessionHandler) java.io.IOException: Connection to 4 was disconnected before the response was read at org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:100) at kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:110) at kafka.server.ReplicaFetcherThread.fetchFromLeader(ReplicaFetcherThread.scala:215) at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:313) at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:139) at kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:138) at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:121) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-12232) Distinguish API scope by broker/controller
[ https://issues.apache.org/jira/browse/KAFKA-12232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-12232. - Resolution: Duplicate > Distinguish API scope by broker/controller > -- > > Key: KAFKA-12232 > URL: https://issues.apache.org/jira/browse/KAFKA-12232 > Project: Kafka > Issue Type: Improvement >Reporter: Jason Gustafson >Priority: Major > > After KIP-500, not all APIs will be available on all listeners. Specifically, > there are controller-only APIs which are only accessible on the controller > listener (e.g. the Raft APIs). In general, we have three API scopes: > client: must be exposed on client listener > broker: must be exposed on inter-broker listener > controller: must be exposed on controller listener > These categories are not mutually exclusive. The `Fetch` API is required on > all listeners as an example, so we need a way to represent the scope as a set > in `ApiKeys`. > We should also put some thought into how this scope is reflected through the > ApiVersions API. I think it makes sense to only advertise APIs that can be > handled. For example, if the controller does not have a handler for the > `FindCoordinator` API, then it doesn't make sense to advertise it. > Potentially we could be even more restrictive when it comes to the > inter-broker APIs. For example, we might not need to advertise > `WriteTxnMarkers` on client-only listeners since a client should never use > this API. Alternatively, we can make it simple and just identify APIs by > controller, broker, or both. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-12278) Keep api versions consistent with api scope
[ https://issues.apache.org/jira/browse/KAFKA-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-12278. - Resolution: Fixed > Keep api versions consistent with api scope > --- > > Key: KAFKA-12278 > URL: https://issues.apache.org/jira/browse/KAFKA-12278 > Project: Kafka > Issue Type: Improvement >Reporter: Jason Gustafson >Assignee: Jason Gustafson >Priority: Major > > With KIP-500, some APIs are only accessible by the broker and some are only > accessible by the controller. We need a better way to indicate the scope of > the API so that we can keep it consistent with the `ApiVersions` API. > Basically we have the following scopes: > - zk broker (e.g. LeaderAndIsr) > - kip-500 broker (e.g. DecommissionBroker) > - kip-500 controller (e.g. Envelope) > These categories are not mutually exclusive. For example, the `Fetch` API > must be exposed in all scopes. We could go even further by distinguishing an > inter-broker scope, but that is probably not needed for now. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-12331) KafkaRaftClient should use the LEO when appending LeaderChangeMessage
[
https://issues.apache.org/jira/browse/KAFKA-12331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Gustafson resolved KAFKA-12331.
-
Resolution: Fixed
> KafkaRaftClient should use the LEO when appending LeaderChangeMessage
> -
>
> Key: KAFKA-12331
> URL: https://issues.apache.org/jira/browse/KAFKA-12331
> Project: Kafka
> Issue Type: Sub-task
> Components: replication
>Reporter: Jose Armando Garcia Sancio
>Assignee: Jose Armando Garcia Sancio
>Priority: Major
>
> KafkaMetadataLog's appendAsLeader expects the base offset to match the LEO.
> This is enforced when KafkaRaftClient uses the BatchAccumulator to create
> batches. When creating the control batch for the LeaderChangeMessage the
> KafkaRaftClient doesn't use the BatchAccumulator and instead creates the
> batch with the default base offset of 0.
> This causes the validation in KafkaMetadataLog to fail with the following
> exception:
> {code:java}
> kafka.common.UnexpectedAppendOffsetException: Unexpected offset in append to
> @metadata-0. First offset 0 is less than the next offset 5. First 10 offsets
> in append: ArrayBuffer(0), last offset in append: 0. Log start offset = 0
> at kafka.log.Log.append(Log.scala:1217)
> at kafka.log.Log.appendAsLeader(Log.scala:1092)
> at kafka.raft.KafkaMetadataLog.appendAsLeader(KafkaMetadataLog.scala:92)
> at
> org.apache.kafka.raft.KafkaRaftClient.appendAsLeader(KafkaRaftClient.java:1158)
> at
> org.apache.kafka.raft.KafkaRaftClient.appendLeaderChangeMessage(KafkaRaftClient.java:449)
> at
> org.apache.kafka.raft.KafkaRaftClient.onBecomeLeader(KafkaRaftClient.java:409)
> at
> org.apache.kafka.raft.KafkaRaftClient.maybeTransitionToLeader(KafkaRaftClient.java:463)
> at
> org.apache.kafka.raft.KafkaRaftClient.handleVoteResponse(KafkaRaftClient.java:663)
> at
> org.apache.kafka.raft.KafkaRaftClient.handleResponse(KafkaRaftClient.java:1530)
> at
> org.apache.kafka.raft.KafkaRaftClient.handleInboundMessage(KafkaRaftClient.java:1652)
> at org.apache.kafka.raft.KafkaRaftClient.poll(KafkaRaftClient.java:2183)
> at kafka.raft.KafkaRaftManager$RaftIoThread.doWork(RaftManager.scala:52)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
> {code}
> We should make the following changes:
> # Fix MockLog to perform similar validation as
> KafkaMetadataLog::appendAsLeader
> # Use the LEO when creating the control batch for the LeaderChangedMessage
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Client/Broker Compatibility Matrix
Feel free to edit the wiki page directly. On 2/18/21 11:19 AM, Gary Russell wrote: > You guys have done a great job enabling mixed client/broker compatibility > since the early days, before 0.10.x.x > > Are there any plans to bring the matrix [1] up-to-date? > > Even just adding "any broker > 1.0.0 - Java: any version" (if that's true) > would be helpful. > > TIA > > [1]: https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix >
[jira] [Created] (KAFKA-12344) Support SlidingWindows in the Scala API
Leah Thomas created KAFKA-12344: --- Summary: Support SlidingWindows in the Scala API Key: KAFKA-12344 URL: https://issues.apache.org/jira/browse/KAFKA-12344 Project: Kafka Issue Type: Improvement Reporter: Leah Thomas Assignee: Leah Thomas in KIP-450 we implemented sliding windows for the Java API but left out a few crucial methods to allow sliding windows to work through the Scala API. We need to add those methods to make the Scala API fully leverage sliding windows -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-12343) Recent change to use SharedTopicAdmin in KakfkaBasedLog fails with AK 0.10.x brokers
Randall Hauch created KAFKA-12343:
-
Summary: Recent change to use SharedTopicAdmin in KakfkaBasedLog
fails with AK 0.10.x brokers
Key: KAFKA-12343
URL: https://issues.apache.org/jira/browse/KAFKA-12343
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Affects Versions: 2.5.2, 2.8.0, 2.7.1, 2.6.2
Reporter: Randall Hauch
Assignee: Randall Hauch
System test failure:
{code:java}
Java.lang.Exception: UnsupportedVersionException: MetadataRequest versions
older than 4 don't support the allowAutoTopicCreation field
at
org.apache.kafka.clients.admin.KafkaAdminClient$Call.fail(KafkaAdminClient.java:755)
at
org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.handleResponses(KafkaAdminClient.java:1136)
at
org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1301)
at
org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1224)
at java.lang.Thread.run(Thread.java:748)
[2021-02-16 12:05:11,735] ERROR [Worker clientId=connect-1,
groupId=connect-cluster] Uncaught exception in herder work thread, exiting:
(org.apache.kafka.connect.runtime.distributed.Di
stributedHerder)
org.apache.kafka.connect.errors.ConnectException: API to get the get the end
offsets for topic 'connect-offsets' is unsupported on brokers at worker25:9092
at
org.apache.kafka.connect.util.TopicAdmin.endOffsets(TopicAdmin.java:680)
at
org.apache.kafka.connect.util.KafkaBasedLog.readToLogEnd(KafkaBasedLog.java:338)
at
org.apache.kafka.connect.util.KafkaBasedLog.start(KafkaBasedLog.java:195)
at
org.apache.kafka.connect.storage.KafkaOffsetBackingStore.start(KafkaOffsetBackingStore.java:136)
at org.apache.kafka.connect.runtime.Worker.start(Worker.java:197)
at
org.apache.kafka.connect.runtime.AbstractHerder.startServices(AbstractHerder.java:128)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:311)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.UnsupportedVersionException: MetadataRequest
versions older than 4 don't support the allowAutoTopicCre
ation field
at
org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at
org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at
org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at
org.apache.kafka.connect.util.TopicAdmin.endOffsets(TopicAdmin.java:668)
... 11 more {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-12342) Get rid of raft/meta log shim layer
Jason Gustafson created KAFKA-12342: --- Summary: Get rid of raft/meta log shim layer Key: KAFKA-12342 URL: https://issues.apache.org/jira/browse/KAFKA-12342 Project: Kafka Issue Type: Improvement Reporter: Jason Gustafson Assignee: Jason Gustafson We currently use a shim to bridge the interface differences between `RaftClient` and `MetaLogManager`. We need to converge the two interfaces and get rid of the shim. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Client/Broker Compatibility Matrix
You guys have done a great job enabling mixed client/broker compatibility since the early days, before 0.10.x.x Are there any plans to bring the matrix [1] up-to-date? Even just adding "any broker > 1.0.0 - Java: any version" (if that's true) would be helpful. TIA [1]: https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix
[jira] [Created] (KAFKA-12341) Duplicate javassist library found in broker classpath
Cosmin Giurgiu created KAFKA-12341:
--
Summary: Duplicate javassist library found in broker classpath
Key: KAFKA-12341
URL: https://issues.apache.org/jira/browse/KAFKA-12341
Project: Kafka
Issue Type: Bug
Components: config
Affects Versions: 2.7.0
Environment: Linux Ubuntu Server 18.04
Reporter: Cosmin Giurgiu
Installed new kafka 2.7.0 with scala 2.13, configured and start broker. Checked
the process information and found:
{code:java}
/opt/jdk11/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20
-XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent
-XX:MaxInlineLevel=15 -Djava.awt.headless=true
-Xlog:gc*:file=/home/kafka/kafka/logs2/kafkaServer-gc.log:time,tags:filecount=10,filesize=100M
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dkafka.logs.dir=/home/kafka/kafka/logs2
-Dlog4j.configuration=file:/home/kafka/kafka/bin/../config/log4j.properties -cp
/home/kafka/kafka/bin/../libs/activation-1.1.1.jar:[..]:/home/kafka/kafka/bin/../libs/javassist-3.25.0-GA.jar:/home/kafka/kafka/bin/../libs/javassist-3.26.0-GA.jar:[..]
kafka.Kafka /home/kafka/kafka/config/server.properties{code}
There are two versions of javassist library - 3.25.0-GA and 3.26.0-GA.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-12340) Recent change to use SharedTopicAdmin results in potential resource leak in deprecated backing store constructors
Randall Hauch created KAFKA-12340: - Summary: Recent change to use SharedTopicAdmin results in potential resource leak in deprecated backing store constructors Key: KAFKA-12340 URL: https://issues.apache.org/jira/browse/KAFKA-12340 Project: Kafka Issue Type: Bug Components: KafkaConnect Affects Versions: 2.5.2, 2.8.0, 2.7.1, 2.6.2 Reporter: Randall Hauch Assignee: Randall Hauch Fix For: 2.5.2, 2.8.0, 2.7.1, 2.6.2 When KAFKA-10021 modified the Connect `Kafka*BackingStore` classes, we deprecated the old constructors and changed all uses within AK to use the new constructors that take a `Supplier`. If the old deprecated constructors are used (outside of AK), then they will not close the Admin clients that are created by the "default" supplier. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-12339) Starting new connector cluster with new internal topics is unstable due to UnknownTopicOrPartitionException
Chia-Ping Tsai created KAFKA-12339:
--
Summary: Starting new connector cluster with new internal topics
is unstable due to UnknownTopicOrPartitionException
Key: KAFKA-12339
URL: https://issues.apache.org/jira/browse/KAFKA-12339
Project: Kafka
Issue Type: Bug
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai
After migrating our connector env to 2.9.0-SNAPSHOT, it start to fail to deploy
connector cluster. The error message is shown below.
{quote}
Uncaught exception in herder work thread, exiting:
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:324)
org.apache.kafka.connect.errors.ConnectException: Error while getting end
offsets for topic 'connect-storage-topic-connect-cluster-1'
at org.apache.kafka.connect.util.TopicAdmin.endOffsets(TopicAdmin.java:689)
at
org.apache.kafka.connect.util.KafkaBasedLog.readToLogEnd(KafkaBasedLog.java:338)
at org.apache.kafka.connect.util.KafkaBasedLog.start(KafkaBasedLog.java:195)
at
org.apache.kafka.connect.storage.KafkaStatusBackingStore.start(KafkaStatusBackingStore.java:216)
at
org.apache.kafka.connect.runtime.AbstractHerder.startServices(AbstractHerder.java:129)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:310)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition.
at
org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at
org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at
org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at org.apache.kafka.connect.util.TopicAdmin.endOffsets(TopicAdmin.java:668)
... 10 more
{quote}
https://github.com/apache/kafka/pull/9780 added shared admin to get end
offsets. KafkaAdmin#listOffsets does not handle topic-level error, hence the
UnknownTopicOrPartitionException on topic-level can obstruct worker from
running when the new internal topic is NOT synced to all brokers.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [VOTE] KIP-516: Topic Identifiers
Hi all, I realized that the DISCUSS thread got very long, so I'll be posting updates to this thread from now on. Just a quick update to the KIP. As a part of https://issues.apache.org/jira/browse/KAFKA-12332 and https://github.com/apache/kafka/pull/10143, I'm proposing adding a new error. INCONSISTENT_TOPIC_ID will be returned on partitions in LeaderAndIsrResponses where the topic ID in the request did not match the topic ID in the log. This will only occur when a valid topic ID is provided in the request. I've also updated the KIP to reflect this change. https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-LeaderAndIsrRequestv5 Please let me know if you have any thoughts or concerns with this change. Thanks, Justine On Mon, Oct 19, 2020 at 8:50 AM Justine Olshan wrote: > Thanks everyone for the votes. KIP-516 has been accepted. > > Binding: Jun, Rajini, David > Non-binding: Lucas, Satish, Tom > > Justine > > On Sat, Oct 17, 2020 at 3:22 AM Tom Bentley wrote: > >> +1 non-binding. Thanks! >> >> On Sat, Oct 17, 2020 at 7:55 AM David Jacot >> wrote: >> >> > Hi Justine, >> > >> > Thanks for the KIP! This is a great and long awaited improvement. >> > >> > +1 (binding) >> > >> > Best, >> > David >> > >> > Le ven. 16 oct. 2020 à 17:36, Rajini Sivaram >> a >> > écrit : >> > >> > > Hi Justine, >> > > >> > > +1 (binding) >> > > >> > > Thanks for all the work you put into this KIP! >> > > >> > > btw, there is a typo in the DeleteTopics Request/Response schema in >> the >> > > KIP, it says Metadata request. >> > > >> > > Regards, >> > > >> > > Rajini >> > > >> > > >> > > On Fri, Oct 16, 2020 at 4:06 PM Satish Duggana < >> satish.dugg...@gmail.com >> > > >> > > wrote: >> > > >> > > > Hi Justine, >> > > > Thanks for the KIP, +1 (non-binding) >> > > > >> > > > On Thu, Oct 15, 2020 at 10:48 PM Lucas Bradstreet < >> lu...@confluent.io> >> > > > wrote: >> > > > > >> > > > > Hi Justine, >> > > > > >> > > > > +1 (non-binding). Thanks for all your hard work on this KIP! >> > > > > >> > > > > Lucas >> > > > > >> > > > > On Wed, Oct 14, 2020 at 8:59 AM Jun Rao wrote: >> > > > > >> > > > > > Hi, Justine, >> > > > > > >> > > > > > Thanks for the updated KIP. +1 from me. >> > > > > > >> > > > > > Jun >> > > > > > >> > > > > > On Tue, Oct 13, 2020 at 2:38 PM Jun Rao >> wrote: >> > > > > > >> > > > > > > Hi, Justine, >> > > > > > > >> > > > > > > Thanks for starting the vote. Just a few minor comments. >> > > > > > > >> > > > > > > 1. It seems that we should remove the topic field from the >> > > > > > > StopReplicaResponse below? >> > > > > > > StopReplica Response (Version: 4) => error_code [topics] >> > > > > > > error_code => INT16 >> > > > > > > topics => topic topic_id* [partitions] >> > > > > > > >> > > > > > > 2. "After controller election, upon receiving the result, >> assign >> > > the >> > > > > > > metadata topic its unique topic ID". Will the UUID for the >> > metadata >> > > > topic >> > > > > > > be written to the metadata topic itself? >> > > > > > > >> > > > > > > 3. The vote request is designed to support multiple topics, >> each >> > of >> > > > them >> > > > > > > may require a different sentinel ID. Should we reserve more >> than >> > > one >> > > > > > > sentinel ID for future usage? >> > > > > > > >> > > > > > > 4. UUID.randomUUID(): Could we clarify whether this method >> > returns >> > > > any >> > > > > > > sentinel ID? Also, how do we expect the user to use it? >> > > > > > > >> > > > > > > Thanks, >> > > > > > > >> > > > > > > Jun >> > > > > > > >> > > > > > > On Mon, Oct 12, 2020 at 9:54 AM Justine Olshan < >> > > jols...@confluent.io >> > > > > >> > > > > > > wrote: >> > > > > > > >> > > > > > >> Hi all, >> > > > > > >> >> > > > > > >> After further discussion and changes to this KIP, I think we >> are >> > > > ready >> > > > > > to >> > > > > > >> restart this vote. >> > > > > > >> >> > > > > > >> Again, here is the KIP: >> > > > > > >> >> > > > > > >> >> > > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers >> > > > > > >> >> > > > > > >> The discussion thread is here: >> > > > > > >> >> > > > > > >> >> > > > > > >> > > > >> > > >> > >> https://lists.apache.org/thread.html/7efa8cd169cadc7dc9cf86a7c0dbbab1836ddb5024d310fcebacf80c@%3Cdev.kafka.apache.org%3E >> > > > > > >> >> > > > > > >> Please take a look and vote if you have a chance. >> > > > > > >> >> > > > > > >> Thanks, >> > > > > > >> Justine >> > > > > > >> >> > > > > > >> On Tue, Sep 22, 2020 at 8:52 AM Justine Olshan < >> > > > jols...@confluent.io> >> > > > > > >> wrote: >> > > > > > >> >> > > > > > >> > Hi all, >> > > > > > >> > >> > > > > > >> > I'd like to call a vote on KIP-516: Topic Identifiers. >> Here is >> > > the >> > > > > > KIP: >> > > > > > >> > >> > > > > > >> > >> > > > > > >> >> > > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers >> > > > >
Re: [DISCUSS] Apache Kafka 2.7.1 release
Thanks for volunteering Mickael. Are there any critical bugs in 2.7.0 so far that motivated the release? There doesn't have to be, but I'd be interested to know if there are. Ismael On Thu, Feb 18, 2021 at 7:58 AM Mickael Maison wrote: > Hi, > > I'd like to volunteer to be the release manager for the next bugfix > release, 2.7.1. > > I created the release plan on the wiki: > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.7.1 > > Thanks >
Re: [DISCUSS] Apache Kafka 2.8.0 release
Hello again, all. This is a notice that we are now in Code Freeze for the 2.8 branch. >From now until the release, only fixes for blockers should be merged to the >release branch. Fixes for failing tests are allowed and encouraged. >Documentation-only commits are also ok, in case you have forgotten to update >the docs for some features in 2.8.0. Once we have a green build and passing system tests, I will cut the first RC. Thank you, John On Sun, Feb 7, 2021, at 09:59, John Roesler wrote: > Hello all, > > I have just cut the branch for 2.8 and sent the notification > email to the dev mailing list. > > As a reminder, the next checkpoint toward the 2.8.0 release > is Code Freeze on Feb 17th. > > To ensure a high-quality release, we should now focus our > efforts on stabilizing the 2.8 branch, including resolving > failures, writing new tests, and fixing documentation. > > Thanks as always for your contributions, > John > > > On Wed, 2021-02-03 at 14:18 -0600, John Roesler wrote: > > Hello again, all, > > > > This is a reminder that today is the Feature Freeze > > deadline. To avoid any last-minute crunch or time-zone > > unfairness, I'll cut the branch toward the end of the week. > > > > Please wrap up your features and transition fully into a > > stabilization mode. The next checkpoint is Code Freeze on > > Feb 17th. > > > > Thanks as always for all of your contributions, > > John > > > > On Wed, 2021-01-27 at 12:17 -0600, John Roesler wrote: > > > Hello again, all. > > > > > > This is a reminder that *today* is the KIP freeze for Apache > > > Kafka 2.8.0. > > > > > > The next checkpoint is the Feature Freeze on Feb 3rd. > > > > > > When considering any last-minute KIPs today, please be > > > mindful of the scope, since we have only one week to merge a > > > stable implementation of the KIP. > > > > > > For those whose KIPs have been accepted already, please work > > > closely with your reviewers so that your features can be > > > merged in a stable form in before the Feb 3rd cutoff. Also, > > > don't forget to update the documentation as part of your > > > feature. > > > > > > Finally, as a gentle reminder to all contributors. There > > > seems to have been a recent increase in test and system test > > > failures. Please take some time starting now to stabilize > > > the codebase so we can ensure a high quality and timely > > > 2.8.0 release! > > > > > > Thanks to all of you for your contributions, > > > John > > > > > > On Sat, 2021-01-23 at 18:15 +0300, Ivan Ponomarev wrote: > > > > Hi John, > > > > > > > > KIP-418 is already implemented and reviewed, but I don't see it in the > > > > release plan. Can it be added? > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-418%3A+A+method-chaining+way+to+branch+KStream > > > > > > > > Regards, > > > > > > > > Ivan > > > > > > > > 22.01.2021 21:49, John Roesler пишет: > > > > > Sure thing, Leah! > > > > > -John > > > > > On Thu, Jan 21, 2021, at 07:54, Leah Thomas wrote: > > > > > > Hi John, > > > > > > > > > > > > KIP-659 was just accepted as well, can it be added to the release > > > > > > plan? > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-659%3A+Improve+TimeWindowedDeserializer+and+TimeWindowedSerde+to+handle+window+size > > > > > > > > > > > > Thanks, > > > > > > Leah > > > > > > > > > > > > On Thu, Jan 14, 2021 at 9:36 AM John Roesler > > > > > > wrote: > > > > > > > > > > > > > Hi David, > > > > > > > > > > > > > > Thanks for the heads-up; it's added. > > > > > > > > > > > > > > -John > > > > > > > > > > > > > > On Thu, 2021-01-14 at 08:43 +0100, David Jacot wrote: > > > > > > > > Hi John, > > > > > > > > > > > > > > > > KIP-700 just got accepted. Can we add it to the release plan? > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-700%3A+Add+Describe+Cluster+API > > > > > > > > > > > > > > > > Thanks, > > > > > > > > David > > > > > > > > > > > > > > > > On Wed, Jan 13, 2021 at 11:22 PM John Roesler > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Thanks, Gary! Sorry for the oversight. > > > > > > > > > -John > > > > > > > > > > > > > > > > > > On Wed, 2021-01-13 at 21:25 +, Gary Russell wrote: > > > > > > > > > > Can you add a link to the summary page [1]? > > > > > > > > > > > > > > > > > > > > I always start there. > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > [1]: > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan > > > > > > > > > > Future release plan - Apache Kafka - Apache Software > > > > > > > > > > Foundation< > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan> > > > > > > > > > > Release Plan 0.10.0; Release Plan 0.10.1; Release Plan > > > > > > > > > > 0.10.2.0; > > > > > > > Release > > > > > > > > > Plan 0.10.2.2; Release Plan 0.11.0.0; Release Plan 0.11.0.3; > > > > > > > >
About Kafka 2.7.0 source code compilation error
Hello! Recently, we need to build a compilation and development environment for Kafka 2.7.0 source code. Now the source code can run successfully. However, when we execute "gradle install" or "gradle build", we will report an error: @ nowarn annotation does not suppress any warnings. The details are as follows: If you delete the @ nowarn annotation according to the location of the error, you will still report other errors, so please find out if there is a good solution, and hope to get your reply. Thank you! The gradle version used is 6.6 The scala version used is 2.13.4
[DISCUSS] Apache Kafka 2.7.1 release
Hi, I'd like to volunteer to be the release manager for the next bugfix release, 2.7.1. I created the release plan on the wiki: https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.7.1 Thanks
Re: [DISCUSS] KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters
Hello everyone, * Sorry, I meant KIP-710. Right now the MirrorMaker cluster is somewhat unreliable, and not supporting running in a cluster properly. I'd say that fixing this would be a nice addition. Does anyone have some input on this? Thanks in advance Daniel Dániel Urbán ezt írta (időpont: 2021. jan. 26., K, 15:56): > Hello everyone, > > I would like to start a discussion on KIP-709, which addresses some > missing features in MM2 dedicated mode. > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-710%3A+Full+support+for+distributed+mode+in+dedicated+MirrorMaker+2.0+clusters > > Currently, the dedicated mode of MM2 does not fully support running in a > cluster. The core issue is that the Connect REST Server is not included in > the dedicated mode, which makes follower->leader communication impossible. > In some cases, this results in the cluster not being able to react to > dynamic configuration changes (e.g. dynamic topic filter changes). > Another smaller detail is that MM2 dedicated mode eagerly resolves config > provider references in the Connector configurations, which is undesirable > and a breaking change compared to vanilla Connect. This can cause an issue > for example when there is an environment variable reference, which contains > some host-specific information, like a file path. The leader resolves the > reference eagerly, and the resolved value is propagated to other MM2 nodes > instead of the reference being resolved locally, separately on each node. > > The KIP addresses these by adding the Connect REST Server to the MM2 > dedicated mode for each replication flow, and postponing the config > provider reference resolution. > > Please discuss, I know this is a major change, but also an important > feature for MM2 users. > > Daniel >
[jira] [Created] (KAFKA-12338) The code of MetadataRecordSerde duplicate with MetadataParser
dengziming created KAFKA-12338: -- Summary: The code of MetadataRecordSerde duplicate with MetadataParser Key: KAFKA-12338 URL: https://issues.apache.org/jira/browse/KAFKA-12338 Project: Kafka Issue Type: Improvement Reporter: dengziming Assignee: dengziming For example: MetadataRecordSerde.recordSize ``` size += ByteUtils.sizeOfUnsignedVarint(data.message().apiKey()); size += ByteUtils.sizeOfUnsignedVarint(data.version()); size += data.message().size(serializationCache, data.version()); ``` MetadataParser.size ``` long messageSize = message.size(cache, version); long totalSize = messageSize + ByteUtils.sizeOfUnsignedVarint(message.apiKey()) + ByteUtils.sizeOfUnsignedVarint(version); ``` we can see that the logic is duplicated except that `MetadataRecordSerde` has an extra `DEFAULT_FRAME_VERSION`, if we want to change the serde format of metadata, we should modify 2 classes, this is unreasonable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-10833) KIP-661: Expose task configurations in Connect REST API
[ https://issues.apache.org/jira/browse/KAFKA-10833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mickael Maison resolved KAFKA-10833. Fix Version/s: 2.8.0 Resolution: Fixed > KIP-661: Expose task configurations in Connect REST API > --- > > Key: KAFKA-10833 > URL: https://issues.apache.org/jira/browse/KAFKA-10833 > Project: Kafka > Issue Type: Improvement >Reporter: Mickael Maison >Assignee: Mickael Maison >Priority: Major > Fix For: 2.8.0 > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-661%3A+Expose+task+configurations+in+Connect+REST+API -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-12303) Flatten SMT drops some fields when null values are present
[
https://issues.apache.org/jira/browse/KAFKA-12303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mickael Maison resolved KAFKA-12303.
Fix Version/s: 3.0.0
Resolution: Fixed
> Flatten SMT drops some fields when null values are present
> --
>
> Key: KAFKA-12303
> URL: https://issues.apache.org/jira/browse/KAFKA-12303
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.0.1, 2.1.1, 2.2.2, 2.3.1, 2.4.1, 2.5.1, 2.7.0, 2.6.1,
> 2.8.0
>Reporter: Chris Egerton
>Assignee: Chris Egerton
>Priority: Major
> Fix For: 3.0.0
>
>
> [This
> line|https://github.com/apache/kafka/blob/0bc394cc1d19f1e41dd6646e9ac0e09b91fb1398/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/Flatten.java#L109]
> should be {{continue}} instead of {{return}}; otherwise, the rest of the
> entries in the currently-being-iterated map are skipped unnecessarily.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-12337) provide full scala api for operators naming
Ramil Israfilov created KAFKA-12337: --- Summary: provide full scala api for operators naming Key: KAFKA-12337 URL: https://issues.apache.org/jira/browse/KAFKA-12337 Project: Kafka Issue Type: Improvement Components: streams Affects Versions: 2.7.0 Reporter: Ramil Israfilov Kafka Streams Java DSL provides possibility to do custom naming for all operators via Named, Grouped, Consumed objects (there is a separate dev guide page about it [https://kafka.apache.org/27/documentation/streams/developer-guide/dsl-topology-naming.html] ) But Scala api for Kafka Streams provide only partial support. For example following API's are missing custom naming: filter,selectKey, map, mapValues, flatMap... Probably there is same issue for other scala objects. As workaround I have to do quite ugly calls to inner KStream java class and perform scala2java and back conversions. Would be really handy if all custom naming API's will be also supported on Scala Kafka Streams DSL -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-12336) custom stream naming does not work while calling stream[K, V](topicPattern: Pattern) API with named Consumed parameter
Ramil Israfilov created KAFKA-12336:
---
Summary: custom stream naming does not work while calling
stream[K, V](topicPattern: Pattern) API with named Consumed parameter
Key: KAFKA-12336
URL: https://issues.apache.org/jira/browse/KAFKA-12336
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 2.7.0
Reporter: Ramil Israfilov
In our Scala application I am trying to implement custom naming for Kafka
Streams application nodes.
We are using topicPattern for our stream source.
Here is an API which I am calling:
{code:java}
val topicsPattern="t-[A-Za-z0-9-].suffix"
val operations: KStream[MyKey, MyValue] =
builder.stream[MyKey, MyValue](Pattern.compile(topicsPattern))(
Consumed.`with`[MyKey, MyValue].withName("my-fancy-name")
)
{code}
Despite the fact that I am providing Consumed with custom name the topology
describe still show "KSTREAM-SOURCE-00" as name for our stream source.
It is not a problem if I just use a name for topic. But our application needs
to get messages from set of topics based on topicname pattern matching.
After checking the kakfa code I see that
org.apache.kafka.streams.kstream.internals.InternalStreamBuilder (on line 103)
has a bug:
{code:java}
public KStream stream(final Pattern topicPattern,
final ConsumedInternal consumed) {
final String name = newProcessorName(KStreamImpl.SOURCE_NAME);
final StreamSourceNode streamPatternSourceNode = new
StreamSourceNode<>(name, topicPattern, consumed);
{code}
node name construction does not take into account the name of consumed
parameter.
For example code for another stream api call with topic name does it correctly:
{code:java}
final String name = new
NamedInternal(consumed.name()).orElseGenerateWithPrefix(this,
KStreamImpl.SOURCE_NAME);
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [DISCUSS] KIP-708: Rack aware Kafka Streams with pluggable StandbyTask assignor
Hi Levani, Thank you for the KIP. Really interesting! Here my comments: 1. To be consistent with the other configs that involve standbys , I would rename standby.task.assignment.awareness -> standby.replicas.awareness 2. I would also rename the prefix instance.tag -> client.tag 3. The following is a question about prefixes in general that maybe somebody else can answer. In the config it says for other prefixes that it is recommended to use the method *Prefix(final String prop) instead of the raw prefix string. Is the plan to make the raw prefix string private in a future release? Should we consider making only the prefix method for this KIP public? 4. Could you provide a mathematical formula instead of Java code for absolute preferred standby task distribution and the other distributtion properties? Could you also add an example for absolute preffered distribution for the computation of the formula similar to what you did for the other properties? 5. Does the order of the tags given for standby.task.assignment.awareness count? You mention it once, but then for the Partially Preferred standby task distribution property it does not seem to be important. 6. In the section about least preferred standby task distribution, you state that "and one [zone] will be reserved for active task". What do you mean by that? All Streams clients will participate in the task assignment of active tasks irrespective of their tags, right? The statement does also not really fit with the example where active stateful task 0_0 is on Node-1, does it? 7. Could you also say some words about how this KIP affects the current HighAvailabilityTaskAssignor? Best, Bruno On 09.02.21 15:54, Levani Kokhreidze wrote: Hello all, I’ve updated KIP-708 [1] to reflect the latest discussion outcomes. I’m looking forward to your feedback. Regards, Levani [1] - https://cwiki.apache.org/confluence/display/KAFKA/KIP-708%3A+Rack+awarness+for+Kafka+Streams On 2. Feb 2021, at 22:03, Levani Kokhreidze wrote: Hi John. Thanks a lot for this detailed analysis! Yes, that is what I had in mind as well. I also like that idea of having “task.assignment.awareness” configuration to tell which instance tags can be used for rack awareness. I may borrow it for this KIP if you don’t mind :) Thanks again John for this discussion, it’s really valuable. I’ll update the proposal and share it once again in this discussion thread. Regards, Levani On 2. Feb 2021, at 18:47, John Roesler mailto:vvcep...@apache.org>> wrote: Hi Levani, 1. Thanks for the details. I figured it must be something like this two-dimensional definition of "rack". It does seem like, if we make the config take a list of tags, we can define the semantics to be that the system will make a best effort to distribute the standbys over each rack dimension. In your example, there are two clusters and three AZs. The example configs would be: Node 1: instance.tag.cluster: K8s_Cluster1 instance.tag.zone: eu-central-1a task.assignment.awareness: cluster,zone Node 2: instance.tag.cluster: K8s_Cluster1 instance.tag.zone: eu-central-1b task.assignment.awareness: cluster,zone Node 3: instance.tag.cluster: K8s_Cluster1 instance.tag.zone: eu-central-1c task.assignment.awareness: cluster,zone Node 4: instance.tag.cluster: K8s_Cluster2 instance.tag.zone: eu-central-1a task.assignment.awareness: cluster,zone Node 5: instance.tag.cluster: K8s_Cluster2 instance.tag.zone: eu-central-1b task.assignment.awareness: cluster,zone Node 6: instance.tag.cluster: K8s_Cluster2 instance.tag.zone: eu-central-1c task.assignment.awareness: cluster,zone Now, if we have a task 0_0 with an active and two replicas, there are three total copies of the task to distribute over: * 6 instances * 2 clusters * 3 zones There is a constraint that we _cannot_ assign two copies of a task to a single instance, but it seems like the default rack awareness would permit us to assign two copies of a task to a rack, if (and only if) the number of copies is greater than the number of racks. So, the assignment we would get is like this: * assigned to three different instances * one copy in each of zone a, b, and c * two copies in one cluster and one in the other cluster For example, we might have 0_0 assigned to: * Node 1 (cluster 1, zone a) * Node 5 (cluster 2, zone b) * Node 3 (cluster 1, zone c) Is that what you were also thinking? Thanks, -John On Tue, Feb 2, 2021, at 02:24, Levani Kokhreidze wrote: Hi John, 1. Main reason was that it seemed easier change compared to having multiple tags assigned to each host. --- Answering your question what use-case I have in mind: Lets say we have two Kubernetes clusters running the same Kafka Streams application. And each Kubernetes cluster is spanned across multiple AZ. So the setup overall looks something like this: K8s_Cluster1 [eu-central-1a, eu-central-1b, eu-central-1c] K8s_Cluster2 [eu-central-1a, eu-central-1b, eu-central-1c] Now, if Kafka
Re: [VOTE] KIP - 405: Kafka Tiered Storage.
Thanks everyone for the votes. This KIP is accepted with +3 binding (Harsha, Mani, and Jun) votes, and +4 non-binding (Kowshik, Ivan, Kamal, and Prateek) votes. On Wed, 17 Feb 2021 at 22:38, Prateek Agarwal wrote: > > +1 (non-binding). > > On Wed, Feb 17, 2021 at 5:53 AM Jun Rao wrote: > > > Hi, Satish, > > > > Thanks for driving the proposal. +1 from me. > > > > Jun > > > > On Tue, Feb 16, 2021 at 12:08 AM Kamal Chandraprakash < > > kamal.chandraprak...@gmail.com> wrote: > > > > > +1 (non-binding) > > > > > > On Tue, Feb 16, 2021 at 11:22 AM Ivan Yurchenko < > > ivan0yurche...@gmail.com> > > > wrote: > > > > > > > Hi, > > > > > > > > Great! > > > > +1 (non-binding) > > > > > > > > Best, > > > > Ivan > > > > > > > > On Mon, 15 Feb 2021 at 22:47, Kowshik Prakasam > > > wrote: > > > > > > > > > +1 (non-binding). Thanks for the excellent KIP! > > > > > > > > > > > > > > > Cheers, > > > > > Kowshik > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Feb 15, 2021 at 2:50 AM Manikumar > > > > > > > wrote: > > > > > > > > > > > Hi Satish, > > > > > > > > > > > > Thanks for driving this KIP. I’m sure there will be a few tweaks as > > > we > > > > > > implement the KIP, but I > > > > > > think KIP is in good shape. > > > > > > > > > > > > I'm +1 (binding). > > > > > > > > > > > > Thanks, > > > > > > Manikumar > > > > > > > > > > > > On Thu, Feb 11, 2021 at 10:57 PM Harsha Chintalapani < > > > ka...@harsha.io> > > > > > > wrote: > > > > > > > > > > > > > +1 (binding). > > > > > > > > > > > > > > Thanks, > > > > > > > Harsha > > > > > > > > > > > > > > On Thu, Feb 11, 2021 at 6:21 AM Satish Duggana < > > sati...@apache.org > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi All, > > > > > > > > We would like to start voting on “KIP-405: Kafka Tiered > > Storage”. > > > > > > > > > > > > > > > > For reference here is the KIP: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Satish. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > Prateek Agarwal
Re: [DISCUSS] KIP-699: Update FindCoordinator to resolve multiple Coordinators at a time
Hi, I have not seen any feedback on this KIP. I'll wait a few more days and if I don't see anything, I'll start a vote early next week. Thanks On Thu, Jan 28, 2021 at 10:43 AM Mickael Maison wrote: > > Hi all, > > I would like to start a discussion on KIP-699: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-699%3A+Update+FindCoordinator+to+resolve+multiple+Coordinators+at+a+time > > This will allow finding coordinators for multiple resources in a > single FindCoordinator request. > > Let me know if you have any feedback. > Thanks
