[GitHub] [kafka] showuon commented on pull request #11616: MINOR: update the description in example properties file
showuon commented on pull request #11616: URL: https://github.com/apache/kafka/pull/11616#issuecomment-997341888 @rondagostino @cmccabe , please help review. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] showuon commented on a change in pull request #11616: MINOR: update the description in example properties file
showuon commented on a change in pull request #11616:
URL: https://github.com/apache/kafka/pull/11616#discussion_r771907372
##
File path: core/src/main/scala/kafka/server/KafkaConfig.scala
##
@@ -712,7 +712,8 @@ object KafkaConfig {
"If it is not set, the metadata log is placed in the first log directory
from log.dirs."
val MetadataSnapshotMaxNewRecordBytesDoc = "This is the maximum number of
bytes in the log between the latest snapshot and the high-watermark needed
before generating a new snapshot."
val ControllerListenerNamesDoc = "A comma-separated list of the names of the
listeners used by the controller. This is required " +
-"if running in KRaft mode. The ZK-based controller will not use this
configuration."
+"if running in KRaft mode. When communicating with the controller quorum,
the broker will always use the first listener in this list.\n " +
+"Note: The ZK-based controller should not set this configuration."
Review comment:
After https://github.com/apache/kafka/pull/11503, a ZooKeeper-based
cluster that sets this config will fail to restart until this config is
removed. We should note it in the doc, not just "ignore the config" as before.
Besides, we should also have simple description explain we will use the 1st
listener to communicating with the controller quorum. (ref:
[KIP-631](https://cwiki.apache.org/confluence/display/KAFKA/KIP-631%3A+The+Quorum-based+Kafka+Controller))
Before
`
A comma-separated list of the names of the listeners used by the controller.
This is required if running in KRaft mode. The ZK-based controller will not use
this configuration.
`
After:
`
A comma-separated list of the names of the listeners used by the controller.
This is required if running in KRaft mode. When communicating with the
controller quorum, the broker will always use the first listener in this list.`
`Note: The ZK-based controller should not set this configuration. `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] showuon commented on a change in pull request #11616: MINOR: update the description in example properties file
showuon commented on a change in pull request #11616:
URL: https://github.com/apache/kafka/pull/11616#discussion_r771907372
##
File path: core/src/main/scala/kafka/server/KafkaConfig.scala
##
@@ -712,7 +712,8 @@ object KafkaConfig {
"If it is not set, the metadata log is placed in the first log directory
from log.dirs."
val MetadataSnapshotMaxNewRecordBytesDoc = "This is the maximum number of
bytes in the log between the latest snapshot and the high-watermark needed
before generating a new snapshot."
val ControllerListenerNamesDoc = "A comma-separated list of the names of the
listeners used by the controller. This is required " +
-"if running in KRaft mode. The ZK-based controller will not use this
configuration."
+"if running in KRaft mode. When communicating with the controller quorum,
the broker will always use the first listener in this list.\n " +
+"Note: The ZK-based controller should not set this configuration."
Review comment:
After https://github.com/apache/kafka/pull/11503, a ZooKeeper-based
cluster that sets this config will fail to restart until this config is
removed. We should note it in the doc, not just "ignore the config" as before.
Besides, we should also have simple description explain we will use the 1st
listener to communicating with the controller quorum. (ref:
[KIP-631](https://cwiki.apache.org/confluence/display/KAFKA/KIP-631%3A+The+Quorum-based+Kafka+Controller))
Before
```
A comma-separated list of the names of the listeners used by the controller.
This is required if running in KRaft mode. The ZK-based controller will not use
this configuration.
```
After:
```
A comma-separated list of the names of the listeners used by the controller.
This is required if running in KRaft mode. When communicating with the
controller quorum, the broker will always use the first listener in this list.
Note: The ZK-based controller should not set this configuration.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] showuon commented on a change in pull request #11616: MINOR: update the description in example properties file
showuon commented on a change in pull request #11616: URL: https://github.com/apache/kafka/pull/11616#discussion_r771906660 ## File path: config/producer.properties ## @@ -23,7 +23,7 @@ bootstrap.servers=localhost:9092 # specify the compression codec for all data generated: none, gzip, snappy, lz4, zstd compression.type=none -# name of the partitioner class for partitioning events; default partition spreads data randomly +# name of the partitioner class for partitioning events; default partitioner will try sticking to a partition until the batch is full #partitioner.class= Review comment: side fix: the default partitioner has changed to sticky partitioner. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] showuon commented on a change in pull request #11616: MINOR: update the description in example properties file
showuon commented on a change in pull request #11616: URL: https://github.com/apache/kafka/pull/11616#discussion_r771906602 ## File path: config/kraft/broker.properties ## @@ -31,22 +31,23 @@ controller.quorum.voters=1@localhost:9093 # Socket Server Settings # -# The address the socket server listens on. It will get the value returned from -# java.net.InetAddress.getCanonicalHostName() if not configured. +# The address the socket server listens on. If not configured, the host name will get from +# java.net.InetAddress.getCanonicalHostName(), with PLAINTEXT listener name, and port 9092. # FORMAT: # listeners = listener_name://host_name:port # EXAMPLE: # listeners = PLAINTEXT://your.host.name:9092 listeners=PLAINTEXT://localhost:9092 + +# Name of listener used for communication between brokers. inter.broker.listener.name=PLAINTEXT -# Hostname and port the broker will advertise to producers and consumers. If not set, -# it uses the value for "listeners" if configured. Otherwise, it will use the value -# returned from java.net.InetAddress.getCanonicalHostName(). +# Listener name, hostname and port the broker will advertise to producers and consumers. +# If not set, it uses the value for "listeners". advertised.listeners=PLAINTEXT://localhost:9092 -# Listener, host name, and port for the controller to advertise to the brokers. If -# this server is a controller, this listener must be configured. +# A comma-separated list of the names of the listeners used by the controller. +# This is required if running in KRaft mode. controller.listener.names=CONTROLLER Review comment: Fix the wrong description for `controller.listener.names` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] showuon commented on a change in pull request #11616: MINOR: update the description in example properties file
showuon commented on a change in pull request #11616: URL: https://github.com/apache/kafka/pull/11616#discussion_r771906571 ## File path: config/kraft/broker.properties ## @@ -31,22 +31,23 @@ controller.quorum.voters=1@localhost:9093 # Socket Server Settings # -# The address the socket server listens on. It will get the value returned from -# java.net.InetAddress.getCanonicalHostName() if not configured. +# The address the socket server listens on. If not configured, the host name will get from +# java.net.InetAddress.getCanonicalHostName(), with PLAINTEXT listener name, and port 9092. # FORMAT: # listeners = listener_name://host_name:port # EXAMPLE: # listeners = PLAINTEXT://your.host.name:9092 listeners=PLAINTEXT://localhost:9092 + +# Name of listener used for communication between brokers. inter.broker.listener.name=PLAINTEXT -# Hostname and port the broker will advertise to producers and consumers. If not set, -# it uses the value for "listeners" if configured. Otherwise, it will use the value -# returned from java.net.InetAddress.getCanonicalHostName(). +# Listener name, hostname and port the broker will advertise to producers and consumers. +# If not set, it uses the value for "listeners". advertised.listeners=PLAINTEXT://localhost:9092 Review comment: Simplify it, to remove the last sentence since it's introduced in `listeners` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] showuon commented on a change in pull request #11616: MINOR: update the description in example properties file
showuon commented on a change in pull request #11616: URL: https://github.com/apache/kafka/pull/11616#discussion_r771906523 ## File path: config/kraft/broker.properties ## @@ -31,22 +31,23 @@ controller.quorum.voters=1@localhost:9093 # Socket Server Settings # -# The address the socket server listens on. It will get the value returned from -# java.net.InetAddress.getCanonicalHostName() if not configured. +# The address the socket server listens on. If not configured, the host name will get from +# java.net.InetAddress.getCanonicalHostName(), with PLAINTEXT listener name, and port 9092. # FORMAT: # listeners = listener_name://host_name:port # EXAMPLE: # listeners = PLAINTEXT://your.host.name:9092 listeners=PLAINTEXT://localhost:9092 + +# Name of listener used for communication between brokers. inter.broker.listener.name=PLAINTEXT -# Hostname and port the broker will advertise to producers and consumers. If not set, -# it uses the value for "listeners" if configured. Otherwise, it will use the value -# returned from java.net.InetAddress.getCanonicalHostName(). +# Listener name, hostname and port the broker will advertise to producers and consumers. +# If not set, it uses the value for "listeners". Review comment: Simplify it, to remove the last sentence since it's introduced in `listeners` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] showuon commented on a change in pull request #11616: MINOR: update the description in example properties file
showuon commented on a change in pull request #11616: URL: https://github.com/apache/kafka/pull/11616#discussion_r771906468 ## File path: config/kraft/broker.properties ## @@ -31,22 +31,23 @@ controller.quorum.voters=1@localhost:9093 # Socket Server Settings # -# The address the socket server listens on. It will get the value returned from -# java.net.InetAddress.getCanonicalHostName() if not configured. +# The address the socket server listens on. If not configured, the host name will get from +# java.net.InetAddress.getCanonicalHostName(), with PLAINTEXT listener name, and port 9092. # FORMAT: # listeners = listener_name://host_name:port # EXAMPLE: # listeners = PLAINTEXT://your.host.name:9092 listeners=PLAINTEXT://localhost:9092 + +# Name of listener used for communication between brokers. inter.broker.listener.name=PLAINTEXT Review comment: It'd be better and clear to have an simple description for each config. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] showuon commented on a change in pull request #11616: MINOR: update the description in example properties file
showuon commented on a change in pull request #11616: URL: https://github.com/apache/kafka/pull/11616#discussion_r771906368 ## File path: config/kraft/broker.properties ## @@ -31,22 +31,23 @@ controller.quorum.voters=1@localhost:9093 # Socket Server Settings # -# The address the socket server listens on. It will get the value returned from -# java.net.InetAddress.getCanonicalHostName() if not configured. +# The address the socket server listens on. If not configured, the host name will get from +# java.net.InetAddress.getCanonicalHostName(), with PLAINTEXT listener name, and port 9092. Review comment: Make it clear about which default value will be used if not set. (i.e. `PLAINTEXT://:9092`) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] showuon commented on a change in pull request #11616: MINOR: update the description in example properties file
showuon commented on a change in pull request #11616: URL: https://github.com/apache/kafka/pull/11616#discussion_r771906368 ## File path: config/kraft/broker.properties ## @@ -31,22 +31,23 @@ controller.quorum.voters=1@localhost:9093 # Socket Server Settings # -# The address the socket server listens on. It will get the value returned from -# java.net.InetAddress.getCanonicalHostName() if not configured. +# The address the socket server listens on. If not configured, the host name will get from +# java.net.InetAddress.getCanonicalHostName(), with PLAINTEXT listener name, and port 9092. Review comment: Make it clear about which default value will be used if not set. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] showuon opened a new pull request #11616: MINOR: update the description in example properties file
showuon opened a new pull request #11616: URL: https://github.com/apache/kafka/pull/11616 When testing Kraft mode, I found the description for `controller.listener.names` is wrong and misleading. Fix the error, and update some other descriptions to make them clearer. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (KAFKA-13546) Explicitly specifying default topic creation groups should not make connector fail
[
https://issues.apache.org/jira/browse/KAFKA-13546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
venkat teki updated KAFKA-13546:
Description:
[KIP-158|https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics]
introduced support for Connect worker to allow source connector configurations
to define topic creation settings.
A new source connector configuration {{topic.creation.groups}} was introduced,
which takes a list of groups.
*Expected behavior*
According to KIP-158, specifying value "default" in {{topic.creation.groups}}
configration should throw a warning, but not let connector fail.
*Actual behavior*
Specifying "default" as one of the topic creation groups will make a connector
fail
was:
[KIP-158|https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics]
introduced support for Connect worker to allow source connector configurations
to define topic creation settings.
A new source connector configuration {{topic.creation.groups}} was introduced,
which takes a list of groups.
*Expected behavior*
According to KIP-158, specifying value "default" in {{topic.creation.groups}}
configration should throw a warning, but not let connector fail.
*Actual behavior*
Specifying "default" will make a connector fail
> Explicitly specifying default topic creation groups should not make connector
> fail
> --
>
> Key: KAFKA-13546
> URL: https://issues.apache.org/jira/browse/KAFKA-13546
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.6.0, 2.7.0, 2.6.1, 2.8.0, 2.7.1, 2.6.2, 2.6.3, 2.7.2,
> 2.8.1
>Reporter: venkat teki
>Assignee: venkat teki
>Priority: Major
>
> [KIP-158|https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics]
> introduced support for Connect worker to allow source connector
> configurations to define topic creation settings.
> A new source connector configuration {{topic.creation.groups}} was
> introduced, which takes a list of groups.
> *Expected behavior*
> According to KIP-158, specifying value "default" in {{topic.creation.groups}}
> configration should throw a warning, but not let connector fail.
> *Actual behavior*
> Specifying "default" as one of the topic creation groups will make a
> connector fail
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [kafka] venkatteki opened a new pull request #11615: KAFKA-13546[WIP]: Do not fail connector if default topic creation group is explicitly specified
venkatteki opened a new pull request #11615: URL: https://github.com/apache/kafka/pull/11615 ### Problem: If user specifies `default` group as part of topic.creation.groups config, that is causing the connector to fail. According to KIP-158, specifying `default` group as part of topic.creation.groups config should throw a warning, but not let connector fail. ### Fix Remove the `default` topic creation group from user input if specified explicitly, and log a warning message ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KAFKA-13526) IQv2: Consider more generic logic for mapping between binary and typed queries
[ https://issues.apache.org/jira/browse/KAFKA-13526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17462055#comment-17462055 ] John Roesler commented on KAFKA-13526: -- Here's an add-on demonstrating lazy deserialization of responses: https://github.com/vvcephei/kafka/pull/1 > IQv2: Consider more generic logic for mapping between binary and typed queries > -- > > Key: KAFKA-13526 > URL: https://issues.apache.org/jira/browse/KAFKA-13526 > Project: Kafka > Issue Type: Sub-task >Reporter: John Roesler >Priority: Major > > Right now, typed queries (like KeyQuery) need to be specially handled and > translated to their binary counterparts (like RawKeyQuery). This happens in > the Metered store layers, where the serdes are known. It is necessary because > lower store layers are only able to handle binary data (because they don't > know the serdes). > This situation is not ideal, since the Metered store layers will grow to host > quite a bit of query handling and translation logic, because the relationship > between typed queries and binary counterparts is not obvious, and because we > can only automatically translate known query types. User-supplied queries and > stores will have to work things out using their a-priori knowledge of the > serdes. > > One suggestion (from [~mjsax] ) is to come up with some kind of generic > "query mapping" API, which the Metered stores would use to translate back and > forth between typed and raw keys and values. Users would be able to supply > their own mappings along with their custom queries. > Another option would be to have the Metered stores attach the serdes to the > query on the way down and then to the result on the way up. Then, the serdes > would be available in the bytes store (as part of the request) and to the > users when they get their results (as part of the response). > Other options may also surface once we start playing with ideas. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (KAFKA-13526) IQv2: Consider more generic logic for mapping between binary and typed queries
[ https://issues.apache.org/jira/browse/KAFKA-13526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17462048#comment-17462048 ] John Roesler commented on KAFKA-13526: -- Here's another attempt, which seems to work better: https://github.com/apache/kafka/pull/11614 > IQv2: Consider more generic logic for mapping between binary and typed queries > -- > > Key: KAFKA-13526 > URL: https://issues.apache.org/jira/browse/KAFKA-13526 > Project: Kafka > Issue Type: Sub-task >Reporter: John Roesler >Priority: Major > > Right now, typed queries (like KeyQuery) need to be specially handled and > translated to their binary counterparts (like RawKeyQuery). This happens in > the Metered store layers, where the serdes are known. It is necessary because > lower store layers are only able to handle binary data (because they don't > know the serdes). > This situation is not ideal, since the Metered store layers will grow to host > quite a bit of query handling and translation logic, because the relationship > between typed queries and binary counterparts is not obvious, and because we > can only automatically translate known query types. User-supplied queries and > stores will have to work things out using their a-priori knowledge of the > serdes. > > One suggestion (from [~mjsax] ) is to come up with some kind of generic > "query mapping" API, which the Metered stores would use to translate back and > forth between typed and raw keys and values. Users would be able to supply > their own mappings along with their custom queries. > Another option would be to have the Metered stores attach the serdes to the > query on the way down and then to the result on the way up. Then, the serdes > would be available in the bytes store (as part of the request) and to the > users when they get their results (as part of the response). > Other options may also surface once we start playing with ideas. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [kafka] vvcephei opened a new pull request #11614: POC: refactor IQv2 to push serdes down instead of transforming in Metered
vvcephei opened a new pull request #11614: URL: https://github.com/apache/kafka/pull/11614 ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KAFKA-13077) Replication failing after unclean shutdown of ZK and all brokers
[
https://issues.apache.org/jira/browse/KAFKA-13077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17461952#comment-17461952
]
Shivakumar commented on KAFKA-13077:
[~junrao] : We have restarted the zk and also Kafka brokers, but we still end
up with the ISR=2 and Leader=2 and other brokers are out of ISR.
This scenario is not getting resolved by waiting or by rolling restart of
brokers.
we are in search of a solution that does not involve data loss by deleting the
data directory.
> Replication failing after unclean shutdown of ZK and all brokers
>
>
> Key: KAFKA-13077
> URL: https://issues.apache.org/jira/browse/KAFKA-13077
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Christopher Auston
>Priority: Minor
>
> I am submitting this in the spirit of what can go wrong when an operator
> violates the constraints Kafka depends on. I don't know if Kafka could or
> should handle this more gracefully. I decided to file this issue because it
> was easy to get the problem I'm reporting with Kubernetes StatefulSets (STS).
> By "easy" I mean that I did not go out of my way to corrupt anything, I just
> was not careful when restarting ZK and brokers.
> I violated the constraints of keeping Zookeeper stable and at least one
> running in-sync replica.
> I am running the bitnami/kafka helm chart on Amazon EKS.
> {quote}% kubectl get po kaf-kafka-0 -ojson |jq .spec.containers'[].image'
> "docker.io/bitnami/kafka:2.8.0-debian-10-r43"
> {quote}
> I started with 3 ZK instances and 3 brokers (both STS). I changed the
> cpu/memory requests on both STS and kubernetes proceeded to restart ZK and
> kafka instances at the same time. If I recall correctly there were some
> crashes and several restarts but eventually all the instances were running
> again. It's possible all ZK nodes and all brokers were unavailable at various
> points.
> The problem I noticed was that two of the brokers were just continually
> spitting out messages like:
> {quote}% kubectl logs kaf-kafka-0 --tail 10
> [2021-07-13 14:26:08,871] INFO [ProducerStateManager
> partition=__transaction_state-0] Loading producer state from snapshot file
> 'SnapshotFile(/bitnami/kafka/data/__transaction_state-0/0001.snapshot,1)'
> (kafka.log.ProducerStateManager)
> [2021-07-13 14:26:08,871] WARN [Log partition=__transaction_state-0,
> dir=/bitnami/kafka/data] *Non-monotonic update of high watermark from
> (offset=2744 segment=[0:1048644]) to (offset=1 segment=[0:169])*
> (kafka.log.Log)
> [2021-07-13 14:26:08,874] INFO [Log partition=__transaction_state-10,
> dir=/bitnami/kafka/data] Truncating to offset 2 (kafka.log.Log)
> [2021-07-13 14:26:08,877] INFO [Log partition=__transaction_state-10,
> dir=/bitnami/kafka/data] Loading producer state till offset 2 with message
> format version 2 (kafka.log.Log)
> [2021-07-13 14:26:08,877] INFO [ProducerStateManager
> partition=__transaction_state-10] Loading producer state from snapshot file
> 'SnapshotFile(/bitnami/kafka/data/__transaction_state-10/0002.snapshot,2)'
> (kafka.log.ProducerStateManager)
> [2021-07-13 14:26:08,877] WARN [Log partition=__transaction_state-10,
> dir=/bitnami/kafka/data] Non-monotonic update of high watermark from
> (offset=2930 segment=[0:1048717]) to (offset=2 segment=[0:338])
> (kafka.log.Log)
> [2021-07-13 14:26:08,880] INFO [Log partition=__transaction_state-20,
> dir=/bitnami/kafka/data] Truncating to offset 1 (kafka.log.Log)
> [2021-07-13 14:26:08,882] INFO [Log partition=__transaction_state-20,
> dir=/bitnami/kafka/data] Loading producer state till offset 1 with message
> format version 2 (kafka.log.Log)
> [2021-07-13 14:26:08,882] INFO [ProducerStateManager
> partition=__transaction_state-20] Loading producer state from snapshot file
> 'SnapshotFile(/bitnami/kafka/data/__transaction_state-20/0001.snapshot,1)'
> (kafka.log.ProducerStateManager)
> [2021-07-13 14:26:08,883] WARN [Log partition=__transaction_state-20,
> dir=/bitnami/kafka/data] Non-monotonic update of high watermark from
> (offset=2956 segment=[0:1048608]) to (offset=1 segment=[0:169])
> (kafka.log.Log)
> {quote}
> If I describe that topic I can see that several partitions have a leader of 2
> and the ISR is just 2 (NOTE I added two more brokers and tried to reassign
> the topic onto brokers 2,3,4 which you can see below). The new brokers also
> spit out the messages about "non-monotonic update" just like the original
> followers. This describe output is from the following day.
> {{% kafka-topics.sh ${=BS} -topic __transaction_state -describe}}
> {{Topic: __transaction_state TopicId: i7bBNCeuQMWl-ZMpzrnMAw PartitionCount:
> 50 ReplicationFactor: 3 Configs:
> compression.type=uncompressed,m
[jira] [Commented] (KAFKA-9366) Upgrade log4j to log4j2
[ https://issues.apache.org/jira/browse/KAFKA-9366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17461807#comment-17461807 ] Dongjin Lee commented on KAFKA-9366: [~tarungoswami] I can't give you any guarantee but, [here|http://home.apache.org/~dongjin/post/apache-kafka-log4j2-support/] is a preview based on Apache Kafka 3.0.0. > Upgrade log4j to log4j2 > --- > > Key: KAFKA-9366 > URL: https://issues.apache.org/jira/browse/KAFKA-9366 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 2.2.0, 2.1.1, 2.3.0, 2.4.0 >Reporter: leibo >Assignee: Dongjin Lee >Priority: Critical > Labels: needs-kip > Fix For: 3.2.0 > > > h2. CVE-2019-17571 Detail > Included in Log4j 1.2 is a SocketServer class that is vulnerable to > deserialization of untrusted data which can be exploited to remotely execute > arbitrary code when combined with a deserialization gadget when listening to > untrusted network traffic for log data. This affects Log4j versions up to 1.2 > up to 1.2.17. > > [https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-17571] > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [kafka] dongjinleekr commented on pull request #7898: KAFKA-9366: Change log4j dependency into log4j2
dongjinleekr commented on pull request #7898: URL: https://github.com/apache/kafka/pull/7898#issuecomment-997166170 All // You can find a preview of Apache Kafka 3.0.0 w/ log4j2 2.16.0 [here](http://home.apache.org/~dongjin/post/apache-kafka-log4j2-support/). @dhruvp-8 I have no plan for 2.7.0 or 2.8.0 but, currently working on 2.8.1 now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
