Re: [VOTE] KIP-760: Minimum value for segment.ms and segment.bytes
Hi all Just trying to bump this thread. Thanks Badai On Sat, Sep 11, 2021 at 1:07 AM Badai Aqrandista wrote: > > Hi all > > I would like to start a vote on KIP-760 > (https://cwiki.apache.org/confluence/display/KAFKA/KIP-760%3A+Minimum+value+for+segment.ms+and+segment.bytes). > > I created this KIP because I have seen so many Kafka brokers crash due > to small segment.ms and/or segment.bytes. > > -- > Thanks, > Badai -- Thanks, Badai
[VOTE] KIP-760: Minimum value for segment.ms and segment.bytes
Hi all I would like to start a vote on KIP-760 (https://cwiki.apache.org/confluence/display/KAFKA/KIP-760%3A+Minimum+value+for+segment.ms+and+segment.bytes). I created this KIP because I have seen so many Kafka brokers crash due to small segment.ms and/or segment.bytes. -- Thanks, Badai
[jira] [Resolved] (KAFKA-13040) Increase minimum value of segment.ms and segment.bytes
[
https://issues.apache.org/jira/browse/KAFKA-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Badai Aqrandista resolved KAFKA-13040.
--
Resolution: Duplicate
> Increase minimum value of segment.ms and segment.bytes
> --
>
> Key: KAFKA-13040
> URL: https://issues.apache.org/jira/browse/KAFKA-13040
> Project: Kafka
> Issue Type: Improvement
> Reporter: Badai Aqrandista
> Assignee: Badai Aqrandista
>Priority: Minor
>
> Raised for KIP-760 (linked).
> Many times, Kafka brokers in production crash with "Too many open files"
> error or "Out of memory" errors because some Kafka topics have a lot of
> segment files as a result of small {{segment.ms}} or {{segment.bytes}}. These
> two configuration can be set by any user who is authorized to create topic or
> modify topic configuration.
> To prevent these two configuration from causing Kafka broker crash, they
> should have a minimum value that is big enough.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (KAFKA-7760) Add broker configuration to set minimum value for segment.bytes and segment.ms
[ https://issues.apache.org/jira/browse/KAFKA-7760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Badai Aqrandista reopened KAFKA-7760: - Reopening issue and making this the main ticket for KIP-760 > Add broker configuration to set minimum value for segment.bytes and segment.ms > -- > > Key: KAFKA-7760 > URL: https://issues.apache.org/jira/browse/KAFKA-7760 > Project: Kafka > Issue Type: Improvement > Reporter: Badai Aqrandista > Assignee: Badai Aqrandista >Priority: Major > Labels: kip, newbie > > If someone set segment.bytes or segment.ms at topic level to a very small > value (e.g. segment.bytes=1000 or segment.ms=1000), Kafka will generate a > very high number of segment files. This can bring down the whole broker due > to hitting the maximum open file (for log) or maximum number of mmap-ed file > (for index). > To prevent that from happening, I would like to suggest adding two new items > to the broker configuration: > * min.topic.segment.bytes, defaults to 1048576: The minimum value for > segment.bytes. When someone sets topic configuration segment.bytes to a value > lower than this, Kafka throws an error INVALID VALUE. > * min.topic.segment.ms, defaults to 360: The minimum value for > segment.ms. When someone sets topic configuration segment.ms to a value lower > than this, Kafka throws an error INVALID VALUE. > Thanks > Badai -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS] KIP-760: Increase minimum value of segment.ms and segment.bytes
Ryanne/James Yes, the proposed values are kinda arbitrary. I just want to prevent users from setting segment.ms to 1 ms or segment.bytes to 1 byte, which will blow the file handles count and crash the broker. Because of that and James' input, I have updated the KIP to add broker configuration to set the minimum value. Every environment will need different minimum values. And they may need to be modified temporarily, like James' use case. Thank you for reviewing the KIP. Regards Badai On Thu, Jul 8, 2021 at 12:09 AM Ryanne Dolan wrote: > > Badai, the KIP makes sense to me, but could you maybe add some evidence to > support your proposed values? I'm sure they come from experience but I > don't think they are obvious and so may appear arbitrary. > > Ryanne > > > On Tue, Jul 6, 2021, 4:23 PM Badai Aqrandista > wrote: > > > Hi all > > > > I have just created KIP-760 > > ( > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-760%3A+Increase+minimum+value+of+segment.ms+and+segment.bytes > > ). > > > > I created this KIP because I have seen so many Kafka brokers crash due > > to small segment.ms and/or segment.bytes. > > > > Please let me know what you think. > > > > -- > > Thanks, > > Badai > > -- Thanks, Badai
Re: [DISCUSS] KIP-760: Increase minimum value of segment.ms and segment.bytes
James Thank you for replying. I originally thought about adding two new broker dynamic config to allow administrator to determine the acceptable minimum: min.topic.segment.ms min.topic.segment.bytes I put this in the "rejected alternatives" section because I think adding two more configs to the broker would be too much trouble. Or is it not? Based on your use case, which is a valid use case, adding these dynamic config would still allow your use case to work, with an additional step of lowering the dynamic config temporarily. What do you think? Regards Badai On Wed, Jul 7, 2021 at 2:06 PM James Cheng wrote: > > Badai, > > Thanks for the KIP. > > We sometimes want to force compaction on a topic. This might be because there > is a bad record in the topic, and we want to force it to get deleted. The way > we do this is, we set segment.ms to a small value and write a record, in > order to force a segment roll. And we also set min.cleanable.dirty.ratio=0, > in order to trigger compaction. It's rare that we need to do it, but it > happens sometimes. This change would make it more difficult to do that. With > this KIP, we would have to write up to 1MB of data before causing the segment > roll, or wait an hour. > > Although come to think of it, if my goal is to trigger compaction, then I can > just write my tombstone a couple thousand times. So maybe this KIP just makes > it slightly more tedious, but doesn't make it impossible. > > Another use case is when we want to truncate a topic, so we set a small > segment size and set retention to almost zero, which will allow Kafka to > delete what is in the topic. For that, though, we could also use > kafka-delete-records.sh, so this KIP would not have impact on that particular > use case. > > -James > > > On Jul 6, 2021, at 2:23 PM, Badai Aqrandista > > wrote: > > > > Hi all > > > > I have just created KIP-760 > > (https://cwiki.apache.org/confluence/display/KAFKA/KIP-760%3A+Increase+minimum+value+of+segment.ms+and+segment.bytes). > > > > I created this KIP because I have seen so many Kafka brokers crash due > > to small segment.ms and/or segment.bytes. > > > > Please let me know what you think. > > > > -- > > Thanks, > > Badai > -- Thanks, Badai
[DISCUSS] KIP-760: Increase minimum value of segment.ms and segment.bytes
Hi all I have just created KIP-760 (https://cwiki.apache.org/confluence/display/KAFKA/KIP-760%3A+Increase+minimum+value+of+segment.ms+and+segment.bytes). I created this KIP because I have seen so many Kafka brokers crash due to small segment.ms and/or segment.bytes. Please let me know what you think. -- Thanks, Badai
[jira] [Resolved] (KAFKA-7760) Add broker configuration to set minimum value for segment.bytes and segment.ms
[ https://issues.apache.org/jira/browse/KAFKA-7760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Badai Aqrandista resolved KAFKA-7760. - Resolution: Duplicate > Add broker configuration to set minimum value for segment.bytes and segment.ms > -- > > Key: KAFKA-7760 > URL: https://issues.apache.org/jira/browse/KAFKA-7760 > Project: Kafka > Issue Type: Improvement > Reporter: Badai Aqrandista >Assignee: Dulvin Witharane >Priority: Major > Labels: kip, newbie > > If someone set segment.bytes or segment.ms at topic level to a very small > value (e.g. segment.bytes=1000 or segment.ms=1000), Kafka will generate a > very high number of segment files. This can bring down the whole broker due > to hitting the maximum open file (for log) or maximum number of mmap-ed file > (for index). > To prevent that from happening, I would like to suggest adding two new items > to the broker configuration: > * min.topic.segment.bytes, defaults to 1048576: The minimum value for > segment.bytes. When someone sets topic configuration segment.bytes to a value > lower than this, Kafka throws an error INVALID VALUE. > * min.topic.segment.ms, defaults to 360: The minimum value for > segment.ms. When someone sets topic configuration segment.ms to a value lower > than this, Kafka throws an error INVALID VALUE. > Thanks > Badai -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-13040) Increase minimum value of segment.ms and segment.bytes
Badai Aqrandista created KAFKA-13040:
Summary: Increase minimum value of segment.ms and segment.bytes
Key: KAFKA-13040
URL: https://issues.apache.org/jira/browse/KAFKA-13040
Project: Kafka
Issue Type: Improvement
Reporter: Badai Aqrandista
Many times, Kafka brokers in production crash with "Too many open files" error
or "Out of memory" errors because some Kafka topics have a lot of segment files
as a result of small {{segment.ms}} or {{segment.bytes}}. These two
configuration can be set by any user who is authorized to create topic or
modify topic configuration.
To prevent these two configuration from causing Kafka broker crash, they should
have a minimum value that is big enough.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-12163) Controller should ensure zkVersion is monotonically increasing when sending UpdateMetadata requests.
Badai Aqrandista created KAFKA-12163:
Summary: Controller should ensure zkVersion is monotonically
increasing when sending UpdateMetadata requests.
Key: KAFKA-12163
URL: https://issues.apache.org/jira/browse/KAFKA-12163
Project: Kafka
Issue Type: Bug
Affects Versions: 2.4.1
Reporter: Badai Aqrandista
When sending UpdateMetadata requests, controller does not currently perform any
check to ensure zkVersion is monotonically increasing. If Zookeeper gets into a
bad state, this can cause Kafka cluster to get into a bad state and possible
data loss as well.
Controller should perform a check to protect the Kafka clusters from getting
into a bad state.
Following shows an example of zkVersion going backward at 2020-12-08
14:10:46,420.
{noformat}
[2020-11-23 00:56:20,315] TRACE [Controller id=1153 epoch=196] Sending
UpdateMetadata request
UpdateMetadataPartitionState(topicName='Execution_CustomsStatus',
partitionIndex=6, controllerEpoch=195, leader=2152, leaderEpoch=210, isr=[2154,
2152, 1153, 1152], zkVersion=535, replicas=[2152, 2154, 1152, 1153],
offlineReplicas=[]) to brokers Set(2153, 1152, 2154, 2151, 1153, 2152, 1154)
for partition Execution_CustomsStatus-6 (state.change.logger)
[2020-11-23 01:15:28,449] TRACE [Controller id=1153 epoch=196] Sending
UpdateMetadata request
UpdateMetadataPartitionState(topicName='Execution_CustomsStatus',
partitionIndex=6, controllerEpoch=195, leader=2152, leaderEpoch=210, isr=[2154,
2152, 1153, 1152], zkVersion=535, replicas=[2152, 2154, 1152, 1153],
offlineReplicas=[]) to brokers Set(1151) for partition
Execution_CustomsStatus-6 (state.change.logger)
[2020-11-24 00:15:17,042] TRACE [Controller id=1153 epoch=196] Sending
UpdateMetadata request
UpdateMetadataPartitionState(topicName='Execution_CustomsStatus',
partitionIndex=6, controllerEpoch=196, leader=2152, leaderEpoch=211, isr=[2154,
2152, 1152], zkVersion=536, replicas=[2152, 2154, 1152, 1153],
offlineReplicas=[1153]) to brokers Set(2153, 1152, 2154, 2151, 1153, 2152,
1154, 1151) for partition Execution_CustomsStatus-6 (state.change.logger)
[2020-12-06 21:53:14,887] TRACE [Controller id=1152 epoch=197] Sending
UpdateMetadata request
UpdateMetadataPartitionState(topicName='Execution_CustomsStatus',
partitionIndex=6, controllerEpoch=197, leader=2154, leaderEpoch=212, isr=[2154,
1152, 1153], zkVersion=538, replicas=[2152, 2154, 1152, 1153],
offlineReplicas=[2152]) to brokers Set(2153, 1152, 2154, 2151, 1153, 1154,
1151) for partition Execution_CustomsStatus-6 (state.change.logger)
[2020-12-06 22:11:43,739] TRACE [Controller id=1152 epoch=197] Sending
UpdateMetadata request
UpdateMetadataPartitionState(topicName='Execution_CustomsStatus',
partitionIndex=6, controllerEpoch=197, leader=2154, leaderEpoch=212, isr=[2154,
1152, 1153], zkVersion=538, replicas=[2152, 2154, 1152, 1153],
offlineReplicas=[]) to brokers Set(2152) for partition
Execution_CustomsStatus-6 (state.change.logger)
[2020-12-06 22:11:43,815] TRACE [Controller id=1152 epoch=197] Sending
UpdateMetadata request
UpdateMetadataPartitionState(topicName='Execution_CustomsStatus',
partitionIndex=6, controllerEpoch=197, leader=2154, leaderEpoch=212, isr=[2154,
1152, 1153], zkVersion=538, replicas=[2152, 2154, 1152, 1153],
offlineReplicas=[]) to brokers Set(2153, 1152, 2154, 2151, 1153, 2152, 1154,
1151) for partition Execution_CustomsStatus-6 (state.change.logger)
[2020-12-06 22:12:12,602] TRACE [Controller id=1152 epoch=197] Sending
UpdateMetadata request
UpdateMetadataPartitionState(topicName='Execution_CustomsStatus',
partitionIndex=6, controllerEpoch=197, leader=2154, leaderEpoch=212, isr=[2154,
1152, 1153, 2152], zkVersion=539, replicas=[2152, 2154, 1152, 1153],
offlineReplicas=[]) to brokers Set(2153, 1152, 2154, 2151, 1153, 2152, 1154,
1151) for partition Execution_CustomsStatus-6 (state.change.logger)
[2020-12-06 22:12:17,019] TRACE [Controller id=1152 epoch=197] Sending
UpdateMetadata request
UpdateMetadataPartitionState(topicName='Execution_CustomsStatus',
partitionIndex=6, controllerEpoch=197, leader=2152, leaderEpoch=213, isr=[2154,
1152, 1153, 2152], zkVersion=540, replicas=[2152, 2154, 1152, 1153],
offlineReplicas=[]) to brokers Set(2153, 1152, 2154, 2151, 1153, 2152, 1154,
1151) for partition Execution_CustomsStatus-6 (state.change.logger)
[2020-12-07 00:08:46,077] TRACE [Controller id=1152 epoch=197] Sending
UpdateMetadata request
UpdateMetadataPartitionState(topicName='Execution_CustomsStatus',
partitionIndex=6, controllerEpoch=197, leader=2152, leaderEpoch=214, isr=[1152,
1153, 2152], zkVersion=541, replicas=[2152, 2154, 1152, 1153],
offlineReplicas=[2154]) to brokers Set(2153, 1152, 2154, 2151, 1153, 2152,
1154, 1151) for partition Execution_CustomsStatus-6 (state.change.logger)
[2020-12-07 00:08:54,790] TRACE [Controller id=1152 epoch=197] Sending
[jira] [Created] (KAFKA-12162) Kafka broker continued to run after failing to create "/brokers/ids/X" znode.
Badai Aqrandista created KAFKA-12162:
Summary: Kafka broker continued to run after failing to create
"/brokers/ids/X" znode.
Key: KAFKA-12162
URL: https://issues.apache.org/jira/browse/KAFKA-12162
Project: Kafka
Issue Type: Bug
Affects Versions: 2.4.1
Reporter: Badai Aqrandista
We found that Kafka broker continued to run after it failed to create
"/brokers/ids/X" znode and still acted as a partition leader. Here's the log
snippet.
{code:java}
[2020-12-08 14:10:25,040] INFO Client successfully logged in.
(org.apache.zookeeper.Login)
[2020-12-08 14:10:25,040] INFO Client will use DIGEST-MD5 as SASL mechanism.
(org.apache.zookeeper.client.ZooKeeperSaslClient)
[2020-12-08 14:10:25,040] INFO Opening socket connection to server
kafkaprzk4.example.com/0.0.0.0:2181. Will attempt to SASL-authenticate using
Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2020-12-08 14:10:25,045] INFO Socket connection established, initiating
session, client: /0.0.0.0:36056, server: kafkaprzk4.example.com/0.0.0.0:2181
(org.apache.zookeeper.ClientCnxn)
[2020-12-08 14:10:25,054] WARN Unable to reconnect to ZooKeeper service,
session 0x5002ed5001b has expired (org.apache.zookeeper.ClientCnxn)
[2020-12-08 14:10:25,054] INFO Unable to reconnect to ZooKeeper service,
session 0x5002ed5001b has expired, closing socket connection
(org.apache.zookeeper.ClientCnxn)
[2020-12-08 14:10:25,055] INFO EventThread shut down for session:
0x5002ed5001b (org.apache.zookeeper.ClientCnxn)
[2020-12-08 14:10:25,055] INFO [ZooKeeperClient Kafka server] Session expired.
(kafka.zookeeper.ZooKeeperClient)
[2020-12-08 14:10:25,059] INFO [ZooKeeperClient Kafka server] Initializing a
new session to
kafkaprzk1.example.com:2181,kafkaprzk2.example.com:2181,kafkaprzk3.example.com:2181,kafkaprzk4.example.com:2181,kafkaprzk5.example.com:2181/kafkaprod/01.
(kafka.zookeeper.ZooKeeperClient)
[2020-12-08 14:10:25,059] INFO Initiating client connection,
connectString=kafkaprzk1.example.com:2181,kafkaprzk2.example.com:2181,kafkaprzk3.example.com:2181,kafkaprzk4.example.com:2181,kafkaprzk5.example.com:2181/kafkaprod/01
sessionTimeout=22500
watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@1b11171f
(org.apache.zookeeper.ZooKeeper)
[2020-12-08 14:10:25,060] INFO jute.maxbuffer value is 4194304 Bytes
(org.apache.zookeeper.ClientCnxnSocket)
[2020-12-08 14:10:25,060] INFO zookeeper.request.timeout value is 0. feature
enabled= (org.apache.zookeeper.ClientCnxn)
[2020-12-08 14:10:25,061] INFO Client successfully logged in.
(org.apache.zookeeper.Login)
[2020-12-08 14:10:25,061] INFO Client will use DIGEST-MD5 as SASL mechanism.
(org.apache.zookeeper.client.ZooKeeperSaslClient)
[2020-12-08 14:10:25,061] INFO Opening socket connection to server
kafkaprzk4.example.com/0.0.0.0:2181. Will attempt to SASL-authenticate using
Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2020-12-08 14:10:25,065] INFO Socket connection established, initiating
session, client: /0.0.0.0:36058, server: kafkaprzk4.example.com/0.0.0.0:2181
(org.apache.zookeeper.ClientCnxn)
[2020-12-08 14:10:25,070] INFO Creating /brokers/ids/2152 (is it secure? true)
(kafka.zk.KafkaZkClient)
[2020-12-08 14:10:25,081] INFO Session establishment complete on server
kafkaprzk4.example.com/0.0.0.0:2181, sessionid = 0x400645dad9d0001, negotiated
timeout = 22500 (org.apache.zookeeper.ClientCnxn)
[2020-12-08 14:10:25,112] ERROR Error while creating ephemeral at
/brokers/ids/2152, node already exists and owner '288459910057232437' does not
match current session '288340729659195393'
(kafka.zk.KafkaZkClient$CheckedEphemeral)
[2020-12-08 14:10:29,814] WARN [Producer clientId=confluent-metrics-reporter]
Got error produce response with correlation id 520478 on topic-partition
_confluent-metrics-9, retrying (9 attempts left). Error:
NOT_LEADER_FOR_PARTITION (org.apache.kafka.clients.producer.internals.Sender)
[2020-12-08 14:10:29,814] WARN [Producer clientId=confluent-metrics-reporter]
Received invalid metadata error in produce request on partition
_confluent-metrics-9 due to
org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is
not the leader for that topic-partition.. Going to request metadata update now
(org.apache.kafka.clients.producer.internals.Sender)
[2020-12-08 14:10:29,818] WARN [Producer clientId=confluent-metrics-reporter]
Got error produce response with correlation id 520480 on topic-partition
_confluent-metrics-4, retrying (9 attempts left). Error:
NOT_LEADER_FOR_PARTITION (org.apache.kafka.clients.producer.internals.Sender)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-10872) Log broker configuration prefixed with "listener.name.*"
Badai Aqrandista created KAFKA-10872:
Summary: Log broker configuration prefixed with "listener.name.*"
Key: KAFKA-10872
URL: https://issues.apache.org/jira/browse/KAFKA-10872
Project: Kafka
Issue Type: Improvement
Reporter: Badai Aqrandista
When configuring broker listeners with "listener.name.*" prefix, it is very
hard to verify in the log if we are passing the correct value or if we're
missing any values.
Can we log these configuration at INFO level? For example:
{code:java}
listener.name.internal.ssl.truststore.location
listener.name.internal.ssl.truststore.password
listener.name.internal.ssl.keystore.location
listener.name.internal.ssl.keystore.password
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-10776) JMX metric RequestsPerSec requires API version to access
Badai Aqrandista created KAFKA-10776:
Summary: JMX metric RequestsPerSec requires API version to access
Key: KAFKA-10776
URL: https://issues.apache.org/jira/browse/KAFKA-10776
Project: Kafka
Issue Type: Bug
Reporter: Badai Aqrandista
JMX metric for
"kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce" seems
to require the API version by adding "version=8" at the end of JMX metric name.
{noformat}
badai@Badai-Aqrandista-MBP15 % bin/kafka-run-class kafka.tools.JmxTool
--jmx-url service:jmx:rmi:///jndi/rmi://127.0.0.1:/jmxrmi -object-name
kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce
Trying to connect to JMX url:
service:jmx:rmi:///jndi/rmi://127.0.0.1:/jmxrmi. No matched attributes for
the queried objects
ArrayBuffer(kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce).
{noformat}
{noformat}
badai@Badai-Aqrandista-MBP15 % bin/kafka-run-class kafka.tools.JmxTool
--jmx-url service:jmx:rmi:///jndi/rmi://127.0.0.1:/jmxrmi -object-name
kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce,version=8
Trying to connect to JMX url:
service:jmx:rmi:///jndi/rmi://127.0.0.1:/jmxrmi.
"time","kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce,version=8:Count","kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce,version=8:EventType","kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce,version=8:FifteenMinuteRate","kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce,version=8:FiveMinuteRate","kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce,version=8:MeanRate","kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce,version=8:OneMinuteRate","kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce,version=8:RateUnit"
1606699872861,6,requests,0.1989162534702867,0.19690572377187665,0.15294943373697226,0.1883600179350904,SECONDS
1606699874859,6,requests,0.19781422719176484,0.19365115842372535,0.14552410136055885,0.1732995824406591,SECONDS
1606699876860,8,requests,0.19781422719176484,0.19365115842372535,0.18505354392836157,0.1732995824406591,SECONDS
1606699878863,8,requests,0.19893436710436987,0.19706180478057453,0.1768573689590503,0.19142554703039305,SECONDS
^C{noformat}
While other JMX metric under RequestMetrics do not require to specify API
version:
{noformat}
badai@Badai-Aqrandista-MBP15 confluent-5.5.1 % bin/kafka-run-class
kafka.tools.JmxTool --jmx-url
service:jmx:rmi:///jndi/rmi://127.0.0.1:/jmxrmi -object-name
kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce Trying to
connect to JMX url: service:jmx:rmi:///jndi/rmi://127.0.0.1:/jmxrmi.
"time","kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce:50thPercentile","kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce:75thPercentile","kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce:95thPercentile","kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce:98thPercentile","kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce:999thPercentile","kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce:99thPercentile","kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce:Count","kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce:Max","kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce:Mean","kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce:Min","kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce:StdDev"
1606700344429,2.0,2.0,4.689,31.2799745,46.0,46.0,72,46.0,2.639,1.0,5.436718033280366
1606700346433,2.0,2.0,4.689,31.2799745,46.0,46.0,72,46.0,2.639,1.0,5.436718033280366
^C
{noformat}
This is definitely not documented here:
[https://kafka.apache.org/documentation/#monitoring]
I think "version=X" part in "RequestsPerSec" is a mistake and should be
removed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-10529) Controller should throttle partition reassignment
Badai Aqrandista created KAFKA-10529: Summary: Controller should throttle partition reassignment Key: KAFKA-10529 URL: https://issues.apache.org/jira/browse/KAFKA-10529 Project: Kafka Issue Type: Improvement Reporter: Badai Aqrandista With [KIP-455|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-455%3A+Create+an+Administrative+API+for+Replica+Reassignment]], reassignment can be triggered via AdminClient API. However, when reassigning a large number of topic partitions at once, this can cause a storm of LeaderAndIsr and UpdateMetadata requests, which can occupy Controller thread for some time. And this prevents Controller from processing other requests. So, Controller should throttle sending LeaderAndIsr request when actioning a reassignment request. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter
Hi all Does anyone have time to review this PR? I have created a PR for KIP-431 against the latest trunk: https://github.com/apache/kafka/pull/9099 Regards Badai On Fri, Jul 10, 2020 at 2:57 PM huxi_2b wrote: > > Badai, > > Replying in this thread. Two possible ways come up into my head: > > 1. CreateTime:1592475472398|key1|3|<0>|h1=v1,h2=v2|value1 > 2. CreateTime:1592475472398|key1|3|offset=0|h1=v1,h2=v2|value1 > > I prefer the option #1 which could be accomplished by a complementary usage > description. What do you think? > > On 2020/06/21 13:39:36, Badai Aqrandista wrote: > > Excellent.> > > > > Would like to hear more feedback from others.> > > > > On Sat, Jun 20, 2020 at 1:27 AM David Jacot wrote:> > > >> > > > Hi Badai,> > > >> > > > Thanks for your reply.> > > >> > > > 2. Yes, that makes sense.> > > >> > > > Best,> > > > David> > > >> > > > On Thu, Jun 18, 2020 at 2:08 PM Badai Aqrandista > > > wrote:> > > >> > > > > David> > > > >> > > > > Thank you for replying> > > > >> > > > > 1. It seems that `print.partition` is already implemented. Do you > > > > confirm?> > > > > BADAI: Yes, you are correct. I have removed it from the KIP.> > > > >> > > > > 2. Will `null.literal` be only used when the value of the message> > > > > is NULL or for any fields? Also, it seems that we print out "null"> > > > > today when the key or the value is empty. Shall we use "null" as> > > > > a default instead of ""?> > > > > BADAI: For any fields. Do you think this is useful?> > > > >> > > > > 3. Could we add a small example of the output in the KIP?> > > > > BADAI: Yes, I have updated the KIP to add a couple of example.> > > > >> > > > > 4. When there are no headers, are we going to print something> > > > > to indicate it to the user? For instance, we print out NO_TIMESTAMP> > > > > where there is no timestamp.> > > > > BADAI: Yes, good idea. I have updated the KIP to print NO_HEADERS.> > > > >> > > > > Thanks> > > > > Badai> > > > >> > > > >> > > > > On Thu, Jun 18, 2020 at 7:25 PM David Jacot wrote:> > > > > >> > > > > > Hi Badai,> > > > > >> > > > > > Thanks for resuming this. I have few small comments:> > > > > >> > > > > > 1. It seems that `print.partition` is already implemented. Do you> > > > > confirm?> > > > > >> > > > > > 2. Will `null.literal` be only used when the value of the message> > > > > > is NULL or for any fields? Also, it seems that we print out "null"> > > > > > today when the key or the value is empty. Shall we use "null" as> > > > > > a default instead of ""?> > > > > >> > > > > > 3. Could we add a small example of the output in the KIP?> > > > > >> > > > > > 4. When there are no headers, are we going to print something> > > > > > to indicate it to the user? For instance, we print out NO_TIMESTAMP> > > > > > where there is no timestamp.> > > > > >> > > > > > Best,> > > > > > David> > > > > >> > > > > > On Wed, Jun 17, 2020 at 4:53 PM Badai Aqrandista > > > > > wrote:> > > > > >> > > > > > > Hi all,> > > > > > >> > > > > > > I have contacted Mateusz separately and he is ok for me to take > > > > > > over> > > > > > > KIP-431:> > > > > > >> > > > > > >> > > > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-431%3A+Support+of+printing+additional+ConsumerRecord+fields+in+DefaultMessageFormatter> > > > > > >> > > > > > > I have updated it a bit. Can anyone give a quick look at it again > > > > > > and> > > > > > > give me some feedback?> > > > > > >> > > > > > > This feature will be very helpful for people supporting Kafka in> > > > > > > operations.> > > > > > >> > > > > > > If it is ready for a vote, please let me know.> > > > > > >> > > > > > > Thanks> > > > > > > Badai> > > > > > >> > > > > > > On Sat, Jun 13, 2020 at 10:59 PM Badai Aqrandista > > > > > > > > > > > > > wrote:> > > > > > > >> > > > > > > > Mateusz> > > > > > > >> > > > > > > > This KIP would be very useful for debugging. But the last > > > > > > > discussion> > > > > > > > is in Feb 2019.> > > > > > > >> > > > > > > > Are you ok if I take over this KIP?> > > > > > > >> > > > > > > > --> > > > > > > > Thanks,> > > > > > > > Badai> > > > > > >> > > > > > >> > > > > > >> > > > > > > --> > > > > > > Thanks,> > > > > > > Badai> > > > > > >> > > > >> > > > >> > > > >> > > > > --> > > > > Thanks,> > > > > Badai> > > > >> > > > > > > > > -- > > > Thanks,> > > Badai> > > -- Thanks, Badai
Re: [DISCUSS] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter
Hi all, I just realised that KIP-431 introduces an incompatible change when an undocumented property "print.partition" is set to "true". I found this because of the system test failure. However, because the incompatibility affects an undocumented property, I do not have migration plan in this KIP. And I have added the following to KIP-431 ( https://cwiki.apache.org/confluence/display/KAFKA/KIP-431%3A+Support+of+printing+additional+ConsumerRecord+fields+in+DefaultMessageFormatter ). - Compatibility, Deprecation, and Migration Plan KIP-431 introduces incompatibility when "print.partition=true" (this property exists in the code before KIP-431 but not documented). Before KIP-431, "kafka-console-consumer" prints the partition as a number after the value for example: "key1|value1|0". After this KIP, "kafka-console-consumer" prints the partition number prefixed with "Partition:" before the key (if printed) and value, for example: "Partition:0|key1|value1" . Because this property was not documented, no migration plan is implemented in KIP-431. The other changes are backward compatible because they do not exist before. Apart from "print.partition=true", if a user does not use any new parameters, then the output of console consumer will look the same as before. - What do you think? Regards Badai On Wed, Aug 5, 2020 at 12:49 AM Badai Aqrandista wrote: > > Hi all > > I have made additional changes to the PR as per review from David Jacot. > Please review. > > Regards > Badai > > On Wed, Jul 29, 2020 at 11:17 PM Badai Aqrandista wrote: >> >> Hi all >> >> I have created a PR for KIP-431 against the latest trunk: >> https://github.com/apache/kafka/pull/9099 >> >> Please review. >> >> Regards >> Badai >> >> On Tue, Jul 21, 2020 at 2:13 AM Matthias J. Sax wrote: >> > >> > Thanks Badai. LGTM. >> > >> > On 7/19/20 4:26 PM, Badai Aqrandista wrote: >> > > Hi all >> > > >> > > I have made a small change to KIP-431 to make it clearer which one is >> > > "Partition" and "Offset". Also I have moved key field to the back, >> > > before the value: >> > > >> > > $ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic >> > > test --from-beginning --property print.partition=true --property >> > > print.key=true --property print.timestamp=true --property >> > > print.offset=true --property print.headers=true --property >> > > key.separator='|' >> > > >> > > CreateTime:1592475472398|Partition:0|Offset:3|h1:v1,h2:v2|key1|value1 >> > > CreateTime:1592475472456|Partition:0|Offset:4|NO_HEADERS|key2|value2 >> > > >> > > Regards >> > > Badai >> > > >> > > On Sun, Jun 21, 2020 at 11:39 PM Badai Aqrandista >> > > wrote: >> > >> >> > >> Excellent. >> > >> >> > >> Would like to hear more feedback from others. >> > >> >> > >> On Sat, Jun 20, 2020 at 1:27 AM David Jacot wrote: >> > >>> >> > >>> Hi Badai, >> > >>> >> > >>> Thanks for your reply. >> > >>> >> > >>> 2. Yes, that makes sense. >> > >>> >> > >>> Best, >> > >>> David >> > >>> >> > >>> On Thu, Jun 18, 2020 at 2:08 PM Badai Aqrandista >> > >>> wrote: >> > >>> >> > >>>> David >> > >>>> >> > >>>> Thank you for replying >> > >>>> >> > >>>> 1. It seems that `print.partition` is already implemented. Do you >> > >>>> confirm? >> > >>>> BADAI: Yes, you are correct. I have removed it from the KIP. >> > >>>> >> > >>>> 2. Will `null.literal` be only used when the value of the message >> > >>>> is NULL or for any fields? Also, it seems that we print out "null" >> > >>>> today when the key or the value is empty. Shall we use "null" as >> > >>>> a default instead of ""? >> > >>>> BADAI: For any fields. Do you think this is useful? >> > >>>> >> > >>>> 3. Could we add a small example of the output in the KIP? >> > >>>> BADAI: Yes, I have updated the KIP to add a couple of example. >>
Re: [DISCUSS] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter
Hi all I have made additional changes to the PR as per review from David Jacot. Please review. Regards Badai On Wed, Jul 29, 2020 at 11:17 PM Badai Aqrandista wrote: > Hi all > > I have created a PR for KIP-431 against the latest trunk: > https://github.com/apache/kafka/pull/9099 > > Please review. > > Regards > Badai > > On Tue, Jul 21, 2020 at 2:13 AM Matthias J. Sax wrote: > > > > Thanks Badai. LGTM. > > > > On 7/19/20 4:26 PM, Badai Aqrandista wrote: > > > Hi all > > > > > > I have made a small change to KIP-431 to make it clearer which one is > > > "Partition" and "Offset". Also I have moved key field to the back, > > > before the value: > > > > > > $ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic > > > test --from-beginning --property print.partition=true --property > > > print.key=true --property print.timestamp=true --property > > > print.offset=true --property print.headers=true --property > > > key.separator='|' > > > > > > CreateTime:1592475472398|Partition:0|Offset:3|h1:v1,h2:v2|key1|value1 > > > CreateTime:1592475472456|Partition:0|Offset:4|NO_HEADERS|key2|value2 > > > > > > Regards > > > Badai > > > > > > On Sun, Jun 21, 2020 at 11:39 PM Badai Aqrandista > wrote: > > >> > > >> Excellent. > > >> > > >> Would like to hear more feedback from others. > > >> > > >> On Sat, Jun 20, 2020 at 1:27 AM David Jacot > wrote: > > >>> > > >>> Hi Badai, > > >>> > > >>> Thanks for your reply. > > >>> > > >>> 2. Yes, that makes sense. > > >>> > > >>> Best, > > >>> David > > >>> > > >>> On Thu, Jun 18, 2020 at 2:08 PM Badai Aqrandista > wrote: > > >>> > > >>>> David > > >>>> > > >>>> Thank you for replying > > >>>> > > >>>> 1. It seems that `print.partition` is already implemented. Do you > confirm? > > >>>> BADAI: Yes, you are correct. I have removed it from the KIP. > > >>>> > > >>>> 2. Will `null.literal` be only used when the value of the message > > >>>> is NULL or for any fields? Also, it seems that we print out "null" > > >>>> today when the key or the value is empty. Shall we use "null" as > > >>>> a default instead of ""? > > >>>> BADAI: For any fields. Do you think this is useful? > > >>>> > > >>>> 3. Could we add a small example of the output in the KIP? > > >>>> BADAI: Yes, I have updated the KIP to add a couple of example. > > >>>> > > >>>> 4. When there are no headers, are we going to print something > > >>>> to indicate it to the user? For instance, we print out NO_TIMESTAMP > > >>>> where there is no timestamp. > > >>>> BADAI: Yes, good idea. I have updated the KIP to print NO_HEADERS. > > >>>> > > >>>> Thanks > > >>>> Badai > > >>>> > > >>>> > > >>>> On Thu, Jun 18, 2020 at 7:25 PM David Jacot > wrote: > > >>>>> > > >>>>> Hi Badai, > > >>>>> > > >>>>> Thanks for resuming this. I have few small comments: > > >>>>> > > >>>>> 1. It seems that `print.partition` is already implemented. Do you > > >>>> confirm? > > >>>>> > > >>>>> 2. Will `null.literal` be only used when the value of the message > > >>>>> is NULL or for any fields? Also, it seems that we print out "null" > > >>>>> today when the key or the value is empty. Shall we use "null" as > > >>>>> a default instead of ""? > > >>>>> > > >>>>> 3. Could we add a small example of the output in the KIP? > > >>>>> > > >>>>> 4. When there are no headers, are we going to print something > > >>>>> to indicate it to the user? For instance, we print out NO_TIMESTAMP > > >>>>> where there is no timestamp. > > >>>>> > > >>>>> Best, > > >>>>> David > > >>>>> > > >>>>> On Wed, Jun 17, 2020 at 4:53 PM Badai Aqrandista < > ba...@confluent.io> > > >>>> wrote: > > >>>>> > > >>>>>> Hi all, > > >>>>>> > > >>>>>> I have contacted Mateusz separately and he is ok for me to take > over > > >>>>>> KIP-431: > > >>>>>> > > >>>>>> > > >>>>>> > > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-431%3A+Support+of+printing+additional+ConsumerRecord+fields+in+DefaultMessageFormatter > > >>>>>> > > >>>>>> I have updated it a bit. Can anyone give a quick look at it again > and > > >>>>>> give me some feedback? > > >>>>>> > > >>>>>> This feature will be very helpful for people supporting Kafka in > > >>>>>> operations. > > >>>>>> > > >>>>>> If it is ready for a vote, please let me know. > > >>>>>> > > >>>>>> Thanks > > >>>>>> Badai > > >>>>>> > > >>>>>> On Sat, Jun 13, 2020 at 10:59 PM Badai Aqrandista < > ba...@confluent.io> > > >>>>>> wrote: > > >>>>>>> > > >>>>>>> Mateusz > > >>>>>>> > > >>>>>>> This KIP would be very useful for debugging. But the last > discussion > > >>>>>>> is in Feb 2019. > > >>>>>>> > > >>>>>>> Are you ok if I take over this KIP? > > >>>>>>> > > >>>>>>> -- > > >>>>>>> Thanks, > > >>>>>>> Badai > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> -- > > >>>>>> Thanks, > > >>>>>> Badai > > >>>>>> > > >>>> > > >>>> > > >>>> > > >>>> -- > > >>>> Thanks, > > >>>> Badai > > >>>> > > >> > > >> > > >> > > >> -- > > >> Thanks, > > >> Badai > > > > > > > > > > > > > > -- > Thanks, > Badai > -- Thanks, Badai
Re: [DISCUSS] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter
Hi all I have created a PR for KIP-431 against the latest trunk: https://github.com/apache/kafka/pull/9099 Please review. Regards Badai On Tue, Jul 21, 2020 at 2:13 AM Matthias J. Sax wrote: > > Thanks Badai. LGTM. > > On 7/19/20 4:26 PM, Badai Aqrandista wrote: > > Hi all > > > > I have made a small change to KIP-431 to make it clearer which one is > > "Partition" and "Offset". Also I have moved key field to the back, > > before the value: > > > > $ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic > > test --from-beginning --property print.partition=true --property > > print.key=true --property print.timestamp=true --property > > print.offset=true --property print.headers=true --property > > key.separator='|' > > > > CreateTime:1592475472398|Partition:0|Offset:3|h1:v1,h2:v2|key1|value1 > > CreateTime:1592475472456|Partition:0|Offset:4|NO_HEADERS|key2|value2 > > > > Regards > > Badai > > > > On Sun, Jun 21, 2020 at 11:39 PM Badai Aqrandista > > wrote: > >> > >> Excellent. > >> > >> Would like to hear more feedback from others. > >> > >> On Sat, Jun 20, 2020 at 1:27 AM David Jacot wrote: > >>> > >>> Hi Badai, > >>> > >>> Thanks for your reply. > >>> > >>> 2. Yes, that makes sense. > >>> > >>> Best, > >>> David > >>> > >>> On Thu, Jun 18, 2020 at 2:08 PM Badai Aqrandista > >>> wrote: > >>> > >>>> David > >>>> > >>>> Thank you for replying > >>>> > >>>> 1. It seems that `print.partition` is already implemented. Do you > >>>> confirm? > >>>> BADAI: Yes, you are correct. I have removed it from the KIP. > >>>> > >>>> 2. Will `null.literal` be only used when the value of the message > >>>> is NULL or for any fields? Also, it seems that we print out "null" > >>>> today when the key or the value is empty. Shall we use "null" as > >>>> a default instead of ""? > >>>> BADAI: For any fields. Do you think this is useful? > >>>> > >>>> 3. Could we add a small example of the output in the KIP? > >>>> BADAI: Yes, I have updated the KIP to add a couple of example. > >>>> > >>>> 4. When there are no headers, are we going to print something > >>>> to indicate it to the user? For instance, we print out NO_TIMESTAMP > >>>> where there is no timestamp. > >>>> BADAI: Yes, good idea. I have updated the KIP to print NO_HEADERS. > >>>> > >>>> Thanks > >>>> Badai > >>>> > >>>> > >>>> On Thu, Jun 18, 2020 at 7:25 PM David Jacot wrote: > >>>>> > >>>>> Hi Badai, > >>>>> > >>>>> Thanks for resuming this. I have few small comments: > >>>>> > >>>>> 1. It seems that `print.partition` is already implemented. Do you > >>>> confirm? > >>>>> > >>>>> 2. Will `null.literal` be only used when the value of the message > >>>>> is NULL or for any fields? Also, it seems that we print out "null" > >>>>> today when the key or the value is empty. Shall we use "null" as > >>>>> a default instead of ""? > >>>>> > >>>>> 3. Could we add a small example of the output in the KIP? > >>>>> > >>>>> 4. When there are no headers, are we going to print something > >>>>> to indicate it to the user? For instance, we print out NO_TIMESTAMP > >>>>> where there is no timestamp. > >>>>> > >>>>> Best, > >>>>> David > >>>>> > >>>>> On Wed, Jun 17, 2020 at 4:53 PM Badai Aqrandista > >>>> wrote: > >>>>> > >>>>>> Hi all, > >>>>>> > >>>>>> I have contacted Mateusz separately and he is ok for me to take over > >>>>>> KIP-431: > >>>>>> > >>>>>> > >>>>>> > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-431%3A+Support+of+printing+additional+ConsumerRecord+fields+in+DefaultMessageFormatter > >>>>>> > >>>>>> I have updated it a bit. Can anyone give a quick look at it again and > >>>>>> give me some feedback? > >>>>>> > >>>>>> This feature will be very helpful for people supporting Kafka in > >>>>>> operations. > >>>>>> > >>>>>> If it is ready for a vote, please let me know. > >>>>>> > >>>>>> Thanks > >>>>>> Badai > >>>>>> > >>>>>> On Sat, Jun 13, 2020 at 10:59 PM Badai Aqrandista > >>>>>> wrote: > >>>>>>> > >>>>>>> Mateusz > >>>>>>> > >>>>>>> This KIP would be very useful for debugging. But the last discussion > >>>>>>> is in Feb 2019. > >>>>>>> > >>>>>>> Are you ok if I take over this KIP? > >>>>>>> > >>>>>>> -- > >>>>>>> Thanks, > >>>>>>> Badai > >>>>>> > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> Thanks, > >>>>>> Badai > >>>>>> > >>>> > >>>> > >>>> > >>>> -- > >>>> Thanks, > >>>> Badai > >>>> > >> > >> > >> > >> -- > >> Thanks, > >> Badai > > > > > > > -- Thanks, Badai
Re: [DISCUSS] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter
Hi all I have made a small change to KIP-431 to make it clearer which one is "Partition" and "Offset". Also I have moved key field to the back, before the value: $ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --property print.partition=true --property print.key=true --property print.timestamp=true --property print.offset=true --property print.headers=true --property key.separator='|' CreateTime:1592475472398|Partition:0|Offset:3|h1:v1,h2:v2|key1|value1 CreateTime:1592475472456|Partition:0|Offset:4|NO_HEADERS|key2|value2 Regards Badai On Sun, Jun 21, 2020 at 11:39 PM Badai Aqrandista wrote: > > Excellent. > > Would like to hear more feedback from others. > > On Sat, Jun 20, 2020 at 1:27 AM David Jacot wrote: > > > > Hi Badai, > > > > Thanks for your reply. > > > > 2. Yes, that makes sense. > > > > Best, > > David > > > > On Thu, Jun 18, 2020 at 2:08 PM Badai Aqrandista wrote: > > > > > David > > > > > > Thank you for replying > > > > > > 1. It seems that `print.partition` is already implemented. Do you confirm? > > > BADAI: Yes, you are correct. I have removed it from the KIP. > > > > > > 2. Will `null.literal` be only used when the value of the message > > > is NULL or for any fields? Also, it seems that we print out "null" > > > today when the key or the value is empty. Shall we use "null" as > > > a default instead of ""? > > > BADAI: For any fields. Do you think this is useful? > > > > > > 3. Could we add a small example of the output in the KIP? > > > BADAI: Yes, I have updated the KIP to add a couple of example. > > > > > > 4. When there are no headers, are we going to print something > > > to indicate it to the user? For instance, we print out NO_TIMESTAMP > > > where there is no timestamp. > > > BADAI: Yes, good idea. I have updated the KIP to print NO_HEADERS. > > > > > > Thanks > > > Badai > > > > > > > > > On Thu, Jun 18, 2020 at 7:25 PM David Jacot wrote: > > > > > > > > Hi Badai, > > > > > > > > Thanks for resuming this. I have few small comments: > > > > > > > > 1. It seems that `print.partition` is already implemented. Do you > > > confirm? > > > > > > > > 2. Will `null.literal` be only used when the value of the message > > > > is NULL or for any fields? Also, it seems that we print out "null" > > > > today when the key or the value is empty. Shall we use "null" as > > > > a default instead of ""? > > > > > > > > 3. Could we add a small example of the output in the KIP? > > > > > > > > 4. When there are no headers, are we going to print something > > > > to indicate it to the user? For instance, we print out NO_TIMESTAMP > > > > where there is no timestamp. > > > > > > > > Best, > > > > David > > > > > > > > On Wed, Jun 17, 2020 at 4:53 PM Badai Aqrandista > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > I have contacted Mateusz separately and he is ok for me to take over > > > > > KIP-431: > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-431%3A+Support+of+printing+additional+ConsumerRecord+fields+in+DefaultMessageFormatter > > > > > > > > > > I have updated it a bit. Can anyone give a quick look at it again and > > > > > give me some feedback? > > > > > > > > > > This feature will be very helpful for people supporting Kafka in > > > > > operations. > > > > > > > > > > If it is ready for a vote, please let me know. > > > > > > > > > > Thanks > > > > > Badai > > > > > > > > > > On Sat, Jun 13, 2020 at 10:59 PM Badai Aqrandista > > > > > wrote: > > > > > > > > > > > > Mateusz > > > > > > > > > > > > This KIP would be very useful for debugging. But the last discussion > > > > > > is in Feb 2019. > > > > > > > > > > > > Are you ok if I take over this KIP? > > > > > > > > > > > > -- > > > > > > Thanks, > > > > > > Badai > > > > > > > > > > > > > > > > > > > > -- > > > > > Thanks, > > > > > Badai > > > > > > > > > > > > > > > > > -- > > > Thanks, > > > Badai > > > > > > > -- > Thanks, > Badai -- Thanks, Badai
Re: [DISCUSS] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter
Hi all The vote for KIP-431 has passed with 3 binding and 1 non-binding +1s, and no objections. Thanks everyone for reviews and feedback. Regards, Badai On Fri, Jul 10, 2020 at 5:22 PM Badai Aqrandista wrote: > > Hu > > I would make it more readable and move the key next to the value like this: > > CreateTime:1592475472398|Partition:3|Offset:0|Headers:h1=v1,h2=v2|key1|value1 > > What do you think? > > In future KIP, we'll add some formatting like David Jacot suggested. > > Regards > Badai > > On Fri, Jul 10, 2020 at 2:57 PM huxi_2b wrote: > > > > Badai, > > > > Replying in this thread. Two possible ways come up into my head: > > > > 1. CreateTime:1592475472398|key1|3|<0>|h1=v1,h2=v2|value1 > > 2. CreateTime:1592475472398|key1|3|offset=0|h1=v1,h2=v2|value1 > > > > I prefer the option #1 which could be accomplished by a complementary usage > > description. What do you think? > > > > On 2020/06/21 13:39:36, Badai Aqrandista wrote: > > > Excellent.> > > > > > > Would like to hear more feedback from others.> > > > > > > On Sat, Jun 20, 2020 at 1:27 AM David Jacot wrote:> > > > >> > > > > Hi Badai,> > > > >> > > > > Thanks for your reply.> > > > >> > > > > 2. Yes, that makes sense.> > > > >> > > > > Best,> > > > > David> > > > >> > > > > On Thu, Jun 18, 2020 at 2:08 PM Badai Aqrandista > > > > wrote:> > > > >> > > > > > David> > > > > >> > > > > > Thank you for replying> > > > > >> > > > > > 1. It seems that `print.partition` is already implemented. Do you > > > > > confirm?> > > > > > BADAI: Yes, you are correct. I have removed it from the KIP.> > > > > >> > > > > > 2. Will `null.literal` be only used when the value of the message> > > > > > is NULL or for any fields? Also, it seems that we print out "null"> > > > > > today when the key or the value is empty. Shall we use "null" as> > > > > > a default instead of ""?> > > > > > BADAI: For any fields. Do you think this is useful?> > > > > >> > > > > > 3. Could we add a small example of the output in the KIP?> > > > > > BADAI: Yes, I have updated the KIP to add a couple of example.> > > > > >> > > > > > 4. When there are no headers, are we going to print something> > > > > > to indicate it to the user? For instance, we print out NO_TIMESTAMP> > > > > > where there is no timestamp.> > > > > > BADAI: Yes, good idea. I have updated the KIP to print NO_HEADERS.> > > > > >> > > > > > Thanks> > > > > > Badai> > > > > >> > > > > >> > > > > > On Thu, Jun 18, 2020 at 7:25 PM David Jacot > > > > > wrote:> > > > > > >> > > > > > > Hi Badai,> > > > > > >> > > > > > > Thanks for resuming this. I have few small comments:> > > > > > >> > > > > > > 1. It seems that `print.partition` is already implemented. Do you> > > > > > confirm?> > > > > > >> > > > > > > 2. Will `null.literal` be only used when the value of the message> > > > > > > is NULL or for any fields? Also, it seems that we print out "null"> > > > > > > today when the key or the value is empty. Shall we use "null" as> > > > > > > a default instead of ""?> > > > > > >> > > > > > > 3. Could we add a small example of the output in the KIP?> > > > > > >> > > > > > > 4. When there are no headers, are we going to print something> > > > > > > to indicate it to the user? For instance, we print out NO_TIMESTAMP> > > > > > > where there is no timestamp.> > > > > > >> > > > > > > Best,> > > > > > > David> > > > > > >> > > > > > > On Wed, Jun 17, 2020 at 4:53 PM Badai Aqrandista > > > > > > > > > > > > wrote:> > > > > > >> > > > > > > > Hi all,&
Re: [DISCUSS] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter
Hu I would make it more readable and move the key next to the value like this: CreateTime:1592475472398|Partition:3|Offset:0|Headers:h1=v1,h2=v2|key1|value1 What do you think? In future KIP, we'll add some formatting like David Jacot suggested. Regards Badai On Fri, Jul 10, 2020 at 2:57 PM huxi_2b wrote: > > Badai, > > Replying in this thread. Two possible ways come up into my head: > > 1. CreateTime:1592475472398|key1|3|<0>|h1=v1,h2=v2|value1 > 2. CreateTime:1592475472398|key1|3|offset=0|h1=v1,h2=v2|value1 > > I prefer the option #1 which could be accomplished by a complementary usage > description. What do you think? > > On 2020/06/21 13:39:36, Badai Aqrandista wrote: > > Excellent.> > > > > Would like to hear more feedback from others.> > > > > On Sat, Jun 20, 2020 at 1:27 AM David Jacot wrote:> > > >> > > > Hi Badai,> > > >> > > > Thanks for your reply.> > > >> > > > 2. Yes, that makes sense.> > > >> > > > Best,> > > > David> > > >> > > > On Thu, Jun 18, 2020 at 2:08 PM Badai Aqrandista > > > wrote:> > > >> > > > > David> > > > >> > > > > Thank you for replying> > > > >> > > > > 1. It seems that `print.partition` is already implemented. Do you > > > > confirm?> > > > > BADAI: Yes, you are correct. I have removed it from the KIP.> > > > >> > > > > 2. Will `null.literal` be only used when the value of the message> > > > > is NULL or for any fields? Also, it seems that we print out "null"> > > > > today when the key or the value is empty. Shall we use "null" as> > > > > a default instead of ""?> > > > > BADAI: For any fields. Do you think this is useful?> > > > >> > > > > 3. Could we add a small example of the output in the KIP?> > > > > BADAI: Yes, I have updated the KIP to add a couple of example.> > > > >> > > > > 4. When there are no headers, are we going to print something> > > > > to indicate it to the user? For instance, we print out NO_TIMESTAMP> > > > > where there is no timestamp.> > > > > BADAI: Yes, good idea. I have updated the KIP to print NO_HEADERS.> > > > >> > > > > Thanks> > > > > Badai> > > > >> > > > >> > > > > On Thu, Jun 18, 2020 at 7:25 PM David Jacot wrote:> > > > > >> > > > > > Hi Badai,> > > > > >> > > > > > Thanks for resuming this. I have few small comments:> > > > > >> > > > > > 1. It seems that `print.partition` is already implemented. Do you> > > > > confirm?> > > > > >> > > > > > 2. Will `null.literal` be only used when the value of the message> > > > > > is NULL or for any fields? Also, it seems that we print out "null"> > > > > > today when the key or the value is empty. Shall we use "null" as> > > > > > a default instead of ""?> > > > > >> > > > > > 3. Could we add a small example of the output in the KIP?> > > > > >> > > > > > 4. When there are no headers, are we going to print something> > > > > > to indicate it to the user? For instance, we print out NO_TIMESTAMP> > > > > > where there is no timestamp.> > > > > >> > > > > > Best,> > > > > > David> > > > > >> > > > > > On Wed, Jun 17, 2020 at 4:53 PM Badai Aqrandista > > > > > wrote:> > > > > >> > > > > > > Hi all,> > > > > > >> > > > > > > I have contacted Mateusz separately and he is ok for me to take > > > > > > over> > > > > > > KIP-431:> > > > > > >> > > > > > >> > > > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-431%3A+Support+of+printing+additional+ConsumerRecord+fields+in+DefaultMessageFormatter> > > > > > >> > > > > > > I have updated it a bit. Can anyone give a quick look at it again > > > > > > and> > > > > > > give me some feedback?> > > > > > >> > > > > > > This feature will be very helpful for people supporting Kafka in> > > > > > > operations.> > > > > > >> > > > > > > If it is ready for a vote, please let me know.> > > > > > >> > > > > > > Thanks> > > > > > > Badai> > > > > > >> > > > > > > On Sat, Jun 13, 2020 at 10:59 PM Badai Aqrandista > > > > > > > > > > > > > wrote:> > > > > > > >> > > > > > > > Mateusz> > > > > > > >> > > > > > > > This KIP would be very useful for debugging. But the last > > > > > > > discussion> > > > > > > > is in Feb 2019.> > > > > > > >> > > > > > > > Are you ok if I take over this KIP?> > > > > > > >> > > > > > > > --> > > > > > > > Thanks,> > > > > > > > Badai> > > > > > >> > > > > > >> > > > > > >> > > > > > > --> > > > > > > Thanks,> > > > > > > Badai> > > > > > >> > > > >> > > > >> > > > >> > > > > --> > > > > Thanks,> > > > > Badai> > > > >> > > > > > > > > -- > > > Thanks,> > > Badai> > > -- Thanks, Badai
Re: [VOTE] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter
Hu No, the property order does not matter. And yes, anything after "<--" is my comment. Do you have any suggestions on making it more readable, while keeping it backward compatible? Can we discuss this in the DISCUSS thread? Regards Badai On Thu, Jul 9, 2020 at 8:05 PM Hu Xi wrote: > > Hi Badai, > > Thanks for the KIP. A quick question from me: > > "CreateTime:1592475472398|key1|3|0|h1=v1,h2=v2|value1<-- offset 3, > partition 0" > > Seems the partition follows the offset. My question is, does the property > order matter? The partition is always printed following the offset no matter > the order for `print.partition=true` and `print.offset=true` is specified. > How do users get aware of the number `3` means the offset instead of the > partition? I am assuming "<-- offset 3, partition 0" is your comment not the > printed words:-) Am I correct? > > > > 发件人: Badai Aqrandista > 发送时间: 2020年7月9日 17:29 > 收件人: dev > 主题: Re: [VOTE] KIP-431: Support of printing additional ConsumerRecord fields > in DefaultMessageFormatter > > David > > Thanks for the vote. That is a good idea. Will start another KIP once > this one is done. > > Regards > Badai > > On Thu, Jul 9, 2020 at 6:46 PM David Jacot wrote: > > > > Hi Badai, > > > > Thanks for the KIP. I think that it is a nice improvement so I am +1 > > (non-binding). > > > > Long term, I wonder if we could adopt a formatting system similar to > > kafkacat. It > > would reduce the number of properties that one has to set and also allow > > more > > powerful formatting. That could be done as a new formatter for instance. > > > > Example: > > kafkacat -b mybroker -t syslog -f 'Topic %t[%p], offset: %o, key: %k, > > payload: %S bytes: %s\n' > > > > Best, > > David > > > > On Wed, Jul 8, 2020 at 12:30 PM Manikumar wrote: > > > > > +1 (binding) > > > > > > Thanks for the KIP. > > > > > > Thanks, > > > Manikumar > > > > > > > > > > > > On Wed, Jul 8, 2020 at 11:47 AM Matthias J. Sax wrote: > > > > > > > +1 (binding) > > > > > > > > -Matthias > > > > > > > > On 7/7/20 7:16 PM, John Roesler wrote: > > > > > Hi Badai, > > > > > > > > > > Thanks for picking this up. I've reviewed the KIP document and > > > > > the threads you linked. I think we may want to make more > > > > > improvements in the future to the printing of headers in particular, > > > > > but this KIP seems like a clear benefit already. I think we can > > > > > take it incrementally. > > > > > > > > > > I'm +1 (binding) > > > > > > > > > > Thanks, > > > > > -John > > > > > > > > > > On Tue, Jul 7, 2020, at 09:57, Badai Aqrandista wrote: > > > > >> Hi all > > > > >> > > > > >> After resurrecting the discussion thread [1] for KIP-431 and have not > > > > >> received any further feedback for 2 weeks, I would like to resurrect > > > > >> the voting thread [2] for KIP-431. > > > > >> > > > > >> I have updated KIP-431 wiki page > > > > >> ( > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-431%3A+Support+of+printing+additional+ConsumerRecord+fields+in+DefaultMessageFormatter > > > > ) > > > > >> to address Ismael's comment on that thread [3]. > > > > >> > > > > >> Does anyone else have other comments or objections about this KIP? > > > > >> > > > > >> [1] > > > > >> > > > > > > > https://lists.apache.org/thread.html/raabf3268ed05931b8a048fce0d6cdf6a326aee4b0d89713d6e6998d6%40%3Cdev.kafka.apache.org%3E > > > > >> > > > > >> [2] > > > > >> > > > > > > > https://lists.apache.org/thread.html/41fff34873184625370f9e76b8d9257f7a9e7892ab616afe64b4f67c%40%3Cdev.kafka.apache.org%3E > > > > >> > > > > >> [3] > > > > >> > > > > > > > https://lists.apache.org/thread.html/99e9cbaad4a0a49b96db104de450c9f488d4b2b03a09b991bcbadbc7%40%3Cdev.kafka.apache.org%3E > > > > >> > > > > >> -- > > > > >> Thanks, > > > > >> Badai > > > > >> > > > > > > > > > > > > > > > -- > Thanks, > Badai -- Thanks, Badai
Re: [VOTE] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter
David Thanks for the vote. That is a good idea. Will start another KIP once this one is done. Regards Badai On Thu, Jul 9, 2020 at 6:46 PM David Jacot wrote: > > Hi Badai, > > Thanks for the KIP. I think that it is a nice improvement so I am +1 > (non-binding). > > Long term, I wonder if we could adopt a formatting system similar to > kafkacat. It > would reduce the number of properties that one has to set and also allow > more > powerful formatting. That could be done as a new formatter for instance. > > Example: > kafkacat -b mybroker -t syslog -f 'Topic %t[%p], offset: %o, key: %k, > payload: %S bytes: %s\n' > > Best, > David > > On Wed, Jul 8, 2020 at 12:30 PM Manikumar wrote: > > > +1 (binding) > > > > Thanks for the KIP. > > > > Thanks, > > Manikumar > > > > > > > > On Wed, Jul 8, 2020 at 11:47 AM Matthias J. Sax wrote: > > > > > +1 (binding) > > > > > > -Matthias > > > > > > On 7/7/20 7:16 PM, John Roesler wrote: > > > > Hi Badai, > > > > > > > > Thanks for picking this up. I've reviewed the KIP document and > > > > the threads you linked. I think we may want to make more > > > > improvements in the future to the printing of headers in particular, > > > > but this KIP seems like a clear benefit already. I think we can > > > > take it incrementally. > > > > > > > > I'm +1 (binding) > > > > > > > > Thanks, > > > > -John > > > > > > > > On Tue, Jul 7, 2020, at 09:57, Badai Aqrandista wrote: > > > >> Hi all > > > >> > > > >> After resurrecting the discussion thread [1] for KIP-431 and have not > > > >> received any further feedback for 2 weeks, I would like to resurrect > > > >> the voting thread [2] for KIP-431. > > > >> > > > >> I have updated KIP-431 wiki page > > > >> ( > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-431%3A+Support+of+printing+additional+ConsumerRecord+fields+in+DefaultMessageFormatter > > > ) > > > >> to address Ismael's comment on that thread [3]. > > > >> > > > >> Does anyone else have other comments or objections about this KIP? > > > >> > > > >> [1] > > > >> > > > > > https://lists.apache.org/thread.html/raabf3268ed05931b8a048fce0d6cdf6a326aee4b0d89713d6e6998d6%40%3Cdev.kafka.apache.org%3E > > > >> > > > >> [2] > > > >> > > > > > https://lists.apache.org/thread.html/41fff34873184625370f9e76b8d9257f7a9e7892ab616afe64b4f67c%40%3Cdev.kafka.apache.org%3E > > > >> > > > >> [3] > > > >> > > > > > https://lists.apache.org/thread.html/99e9cbaad4a0a49b96db104de450c9f488d4b2b03a09b991bcbadbc7%40%3Cdev.kafka.apache.org%3E > > > >> > > > >> -- > > > >> Thanks, > > > >> Badai > > > >> > > > > > > > > -- Thanks, Badai
Re: [VOTE] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter
Hi all After resurrecting the discussion thread [1] for KIP-431 and have not received any further feedback for 2 weeks, I would like to resurrect the voting thread [2] for KIP-431. I have updated KIP-431 wiki page (https://cwiki.apache.org/confluence/display/KAFKA/KIP-431%3A+Support+of+printing+additional+ConsumerRecord+fields+in+DefaultMessageFormatter) to address Ismael's comment on that thread [3]. Does anyone else have other comments or objections about this KIP? [1] https://lists.apache.org/thread.html/raabf3268ed05931b8a048fce0d6cdf6a326aee4b0d89713d6e6998d6%40%3Cdev.kafka.apache.org%3E [2] https://lists.apache.org/thread.html/41fff34873184625370f9e76b8d9257f7a9e7892ab616afe64b4f67c%40%3Cdev.kafka.apache.org%3E [3] https://lists.apache.org/thread.html/99e9cbaad4a0a49b96db104de450c9f488d4b2b03a09b991bcbadbc7%40%3Cdev.kafka.apache.org%3E -- Thanks, Badai
Re: [DISCUSS] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter
Excellent. Would like to hear more feedback from others. On Sat, Jun 20, 2020 at 1:27 AM David Jacot wrote: > > Hi Badai, > > Thanks for your reply. > > 2. Yes, that makes sense. > > Best, > David > > On Thu, Jun 18, 2020 at 2:08 PM Badai Aqrandista wrote: > > > David > > > > Thank you for replying > > > > 1. It seems that `print.partition` is already implemented. Do you confirm? > > BADAI: Yes, you are correct. I have removed it from the KIP. > > > > 2. Will `null.literal` be only used when the value of the message > > is NULL or for any fields? Also, it seems that we print out "null" > > today when the key or the value is empty. Shall we use "null" as > > a default instead of ""? > > BADAI: For any fields. Do you think this is useful? > > > > 3. Could we add a small example of the output in the KIP? > > BADAI: Yes, I have updated the KIP to add a couple of example. > > > > 4. When there are no headers, are we going to print something > > to indicate it to the user? For instance, we print out NO_TIMESTAMP > > where there is no timestamp. > > BADAI: Yes, good idea. I have updated the KIP to print NO_HEADERS. > > > > Thanks > > Badai > > > > > > On Thu, Jun 18, 2020 at 7:25 PM David Jacot wrote: > > > > > > Hi Badai, > > > > > > Thanks for resuming this. I have few small comments: > > > > > > 1. It seems that `print.partition` is already implemented. Do you > > confirm? > > > > > > 2. Will `null.literal` be only used when the value of the message > > > is NULL or for any fields? Also, it seems that we print out "null" > > > today when the key or the value is empty. Shall we use "null" as > > > a default instead of ""? > > > > > > 3. Could we add a small example of the output in the KIP? > > > > > > 4. When there are no headers, are we going to print something > > > to indicate it to the user? For instance, we print out NO_TIMESTAMP > > > where there is no timestamp. > > > > > > Best, > > > David > > > > > > On Wed, Jun 17, 2020 at 4:53 PM Badai Aqrandista > > wrote: > > > > > > > Hi all, > > > > > > > > I have contacted Mateusz separately and he is ok for me to take over > > > > KIP-431: > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-431%3A+Support+of+printing+additional+ConsumerRecord+fields+in+DefaultMessageFormatter > > > > > > > > I have updated it a bit. Can anyone give a quick look at it again and > > > > give me some feedback? > > > > > > > > This feature will be very helpful for people supporting Kafka in > > > > operations. > > > > > > > > If it is ready for a vote, please let me know. > > > > > > > > Thanks > > > > Badai > > > > > > > > On Sat, Jun 13, 2020 at 10:59 PM Badai Aqrandista > > > > wrote: > > > > > > > > > > Mateusz > > > > > > > > > > This KIP would be very useful for debugging. But the last discussion > > > > > is in Feb 2019. > > > > > > > > > > Are you ok if I take over this KIP? > > > > > > > > > > -- > > > > > Thanks, > > > > > Badai > > > > > > > > > > > > > > > > -- > > > > Thanks, > > > > Badai > > > > > > > > > > > > -- > > Thanks, > > Badai > > -- Thanks, Badai
Re: [DISCUSS] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter
David Thank you for replying 1. It seems that `print.partition` is already implemented. Do you confirm? BADAI: Yes, you are correct. I have removed it from the KIP. 2. Will `null.literal` be only used when the value of the message is NULL or for any fields? Also, it seems that we print out "null" today when the key or the value is empty. Shall we use "null" as a default instead of ""? BADAI: For any fields. Do you think this is useful? 3. Could we add a small example of the output in the KIP? BADAI: Yes, I have updated the KIP to add a couple of example. 4. When there are no headers, are we going to print something to indicate it to the user? For instance, we print out NO_TIMESTAMP where there is no timestamp. BADAI: Yes, good idea. I have updated the KIP to print NO_HEADERS. Thanks Badai On Thu, Jun 18, 2020 at 7:25 PM David Jacot wrote: > > Hi Badai, > > Thanks for resuming this. I have few small comments: > > 1. It seems that `print.partition` is already implemented. Do you confirm? > > 2. Will `null.literal` be only used when the value of the message > is NULL or for any fields? Also, it seems that we print out "null" > today when the key or the value is empty. Shall we use "null" as > a default instead of ""? > > 3. Could we add a small example of the output in the KIP? > > 4. When there are no headers, are we going to print something > to indicate it to the user? For instance, we print out NO_TIMESTAMP > where there is no timestamp. > > Best, > David > > On Wed, Jun 17, 2020 at 4:53 PM Badai Aqrandista wrote: > > > Hi all, > > > > I have contacted Mateusz separately and he is ok for me to take over > > KIP-431: > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-431%3A+Support+of+printing+additional+ConsumerRecord+fields+in+DefaultMessageFormatter > > > > I have updated it a bit. Can anyone give a quick look at it again and > > give me some feedback? > > > > This feature will be very helpful for people supporting Kafka in > > operations. > > > > If it is ready for a vote, please let me know. > > > > Thanks > > Badai > > > > On Sat, Jun 13, 2020 at 10:59 PM Badai Aqrandista > > wrote: > > > > > > Mateusz > > > > > > This KIP would be very useful for debugging. But the last discussion > > > is in Feb 2019. > > > > > > Are you ok if I take over this KIP? > > > > > > -- > > > Thanks, > > > Badai > > > > > > > > -- > > Thanks, > > Badai > > -- Thanks, Badai
Re: [DISCUSS] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter
Hi all, I have contacted Mateusz separately and he is ok for me to take over KIP-431: https://cwiki.apache.org/confluence/display/KAFKA/KIP-431%3A+Support+of+printing+additional+ConsumerRecord+fields+in+DefaultMessageFormatter I have updated it a bit. Can anyone give a quick look at it again and give me some feedback? This feature will be very helpful for people supporting Kafka in operations. If it is ready for a vote, please let me know. Thanks Badai On Sat, Jun 13, 2020 at 10:59 PM Badai Aqrandista wrote: > > Mateusz > > This KIP would be very useful for debugging. But the last discussion > is in Feb 2019. > > Are you ok if I take over this KIP? > > -- > Thanks, > Badai -- Thanks, Badai
Re: [DISCUSS] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter
Mateusz This KIP would be very useful for debugging. But the last discussion is in Feb 2019. Are you ok if I take over this KIP? -- Thanks, Badai
Re: [DISCUSS] KIP-602 - Change default value for client.dns.lookup
Ismael/Rajini I have updated the KIP to reflect the deprecation of "default" value and not adding "use_first_dns_ip". Also I have updated the PR to reflect this change, including changing all references of ClientDnsLookup.DEFAULT to ClientDnsLookup.USE_ALL_DNS_IPS in core code and clients test code. Please let me know what you think. Thanks Badai On Fri, May 29, 2020 at 3:46 AM Ismael Juma wrote: > +1. I think we should remove this config in AK 3.0, but in the meantime, we > can log a warning if people explicitly set the value to `default`. I think > this would be pretty rare. > > Ismael > > On Thu, May 28, 2020 at 10:25 AM Rajini Sivaram > wrote: > > > It is a bit confusing to have a `default` value that is not the default > and > > in hindsight it wasn't a good choice of name. But agree that changing the > > config default and avoiding the temporary `use_first_dns_ip` option makes > > sense. > > > > Regards, > > > > Rajini > > > > > > On Thu, May 28, 2020 at 4:49 PM Badai Aqrandista > > wrote: > > > > > Ismael/Rajini > > > > > > I have put some comments in the PR in response to Ismael's. > > > > > > I have some questions about Ismael's suggestion to not add > > > "use_first_dns_ip" at all and instead just deprecate "default". > > > > > > The PR would be much cleaner if we just deprecate "default" as you > > > suggested. But we will need to update some core code. And I will need > to > > > update the KIP to reflect this. > > > > > > ClientDnsLookup.DEFAULT is used in few places in core (server): > > > > > > > > > > > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaServer.scala#L520 > > > > > > > > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/ReplicaFetcherBlockingSend.scala#L92 > > > > > > > > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/controller/ControllerChannelManager.scala#L156 > > > > > > > > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/coordinator/transaction/TransactionMarkerChannelManager.scala#L82 > > > > > > And a couple of tools: > > > > > > > > > > > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala#L482 > > > > > > > > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/BrokerApiVersionsCommand.scala#L299 > > > > > > And some tests. > > > > > > What do you think? > > > > > > Thanks > > > Badai > > > > > > On Fri, May 22, 2020 at 6:45 PM Badai Aqrandista > > > wrote: > > > > > > > Voting thread has been posted. > > > > > > > > KIP-602 page has been updated with suggestions from Rajini. > > > > > > > > Thanks > > > > Badai > > > > > > > > On Fri, May 22, 2020 at 6:00 AM Ismael Juma > wrote: > > > > > > > >> Badai, would you like to start a vote on this KIP? > > > >> > > > >> Ismael > > > >> > > > >> On Wed, May 20, 2020 at 7:45 AM Rajini Sivaram < > > rajinisiva...@gmail.com > > > > > > > >> wrote: > > > >> > > > >> > Deprecating for removal in 3.0 sounds good. > > > >> > > > > >> > On Wed, May 20, 2020 at 3:33 PM Ismael Juma > > > wrote: > > > >> > > > > >> > > Is there any reason to use "use_first_dns_ip"? Should we remove > it > > > >> > > completely? Or at least deprecate it for removal in 3.0? > > > >> > > > > > >> > > Ismael > > > >> > > > > > >> > > > > > >> > > On Wed, May 20, 2020, 1:39 AM Rajini Sivaram < > > > rajinisiva...@gmail.com > > > >> > > > > >> > > wrote: > > > >> > > > > > >> > > > Hi Badai, > > > >> > > > > > > >> > > > Thanks for the KIP, sounds like a useful change. Perhaps we > > should > > > >> call > > > >> > > the > > > >> > > > new option `use_first_dns_ip` (not `_ips` since it refers to > > one). > > > >
Re: [DISCUSS] KIP-602 - Change default value for client.dns.lookup
Ismael/Rajini I have put some comments in the PR in response to Ismael's. I have some questions about Ismael's suggestion to not add "use_first_dns_ip" at all and instead just deprecate "default". The PR would be much cleaner if we just deprecate "default" as you suggested. But we will need to update some core code. And I will need to update the KIP to reflect this. ClientDnsLookup.DEFAULT is used in few places in core (server): https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaServer.scala#L520 https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/ReplicaFetcherBlockingSend.scala#L92 https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/controller/ControllerChannelManager.scala#L156 https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/coordinator/transaction/TransactionMarkerChannelManager.scala#L82 And a couple of tools: https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala#L482 https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/BrokerApiVersionsCommand.scala#L299 And some tests. What do you think? Thanks Badai On Fri, May 22, 2020 at 6:45 PM Badai Aqrandista wrote: > Voting thread has been posted. > > KIP-602 page has been updated with suggestions from Rajini. > > Thanks > Badai > > On Fri, May 22, 2020 at 6:00 AM Ismael Juma wrote: > >> Badai, would you like to start a vote on this KIP? >> >> Ismael >> >> On Wed, May 20, 2020 at 7:45 AM Rajini Sivaram >> wrote: >> >> > Deprecating for removal in 3.0 sounds good. >> > >> > On Wed, May 20, 2020 at 3:33 PM Ismael Juma wrote: >> > >> > > Is there any reason to use "use_first_dns_ip"? Should we remove it >> > > completely? Or at least deprecate it for removal in 3.0? >> > > >> > > Ismael >> > > >> > > >> > > On Wed, May 20, 2020, 1:39 AM Rajini Sivaram > > >> > > wrote: >> > > >> > > > Hi Badai, >> > > > >> > > > Thanks for the KIP, sounds like a useful change. Perhaps we should >> call >> > > the >> > > > new option `use_first_dns_ip` (not `_ips` since it refers to one). >> We >> > > > should also mention in the KIP that only one type of address (ipv4 >> or >> > > ipv6, >> > > > based on the first one) will be used - that is the current behaviour >> > for >> > > > `use_all_dns_ips`. Since we are changing `default` to be exactly >> the >> > > same >> > > > as `use_all_dns_ips`, it will be good to mention that explicitly >> under >> > > > Public Interfaces. >> > > > >> > > > Regards, >> > > > >> > > > Rajini >> > > > >> > > > >> > > > On Mon, May 18, 2020 at 1:44 AM Badai Aqrandista < >> ba...@confluent.io> >> > > > wrote: >> > > > >> > > > > Ismael >> > > > > >> > > > > What do you think of the PR and the explanation regarding the >> issue >> > > > raised >> > > > > in KIP-235? >> > > > > >> > > > > Should I go ahead and build a proper PR? >> > > > > >> > > > > Thanks >> > > > > Badai >> > > > > >> > > > > On Mon, May 11, 2020 at 8:53 AM Badai Aqrandista < >> ba...@confluent.io >> > > >> > > > > wrote: >> > > > > >> > > > > > Ismael >> > > > > > >> > > > > > PR created: https://github.com/apache/kafka/pull/8644/files >> > > > > > >> > > > > > Also, as this is my first PR, please let me know if I missed >> > > anything. >> > > > > > >> > > > > > Thanks >> > > > > > Badai >> > > > > > >> > > > > > On Mon, May 11, 2020 at 8:19 AM Badai Aqrandista < >> > ba...@confluent.io >> > > > >> > > > > > wrote: >> > > > > > >> > > > > >> Ismael >> > > > > >> >> > > > > >> Thank you for responding. >> > > > > >> >> > > > > >> KIP-235 modified ClientUtils#parseAndValidateAddresses [1] to >> > > resolve &g
Re: [VOTE] KIP-602: Change default value for client.dns.lookup
The vote for KIP-602 has passed with 5 binding and 1 non-binding +1s, and no objections. Thanks everyone for reviews and feedback, Badai On Mon, May 25, 2020 at 12:21 PM Gwen Shapira wrote: > +1 (binding) > > Thank you! > > On Fri, May 22, 2020 at 1:40 AM Badai Aqrandista > wrote: > > > Hi All, > > > > I would like to start the vote on KIP-602: Change default value for > > client.dns.lookup > > > > For reference, here is the KIP wiki: > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-602%3A+Change+default+value+for+client.dns.lookup > > > > And discussion thread: > > > > > > > https://lists.apache.org/thread.html/r0e70d3757267c4158f12c05a4e5ac9eb33f2d11ce99d5878b3b4b3f7%40%3Cdev.kafka.apache.org%3E > > > > -- > > Thanks, > > Badai > > > > > -- > Gwen Shapira > Engineering Manager | Confluent > 650.450.2760 | @gwenshap > Follow us: Twitter | blog > -- Thanks, Badai
Re: [DISCUSS] KIP-602 - Change default value for client.dns.lookup
Voting thread has been posted. KIP-602 page has been updated with suggestions from Rajini. Thanks Badai On Fri, May 22, 2020 at 6:00 AM Ismael Juma wrote: > Badai, would you like to start a vote on this KIP? > > Ismael > > On Wed, May 20, 2020 at 7:45 AM Rajini Sivaram > wrote: > > > Deprecating for removal in 3.0 sounds good. > > > > On Wed, May 20, 2020 at 3:33 PM Ismael Juma wrote: > > > > > Is there any reason to use "use_first_dns_ip"? Should we remove it > > > completely? Or at least deprecate it for removal in 3.0? > > > > > > Ismael > > > > > > > > > On Wed, May 20, 2020, 1:39 AM Rajini Sivaram > > > wrote: > > > > > > > Hi Badai, > > > > > > > > Thanks for the KIP, sounds like a useful change. Perhaps we should > call > > > the > > > > new option `use_first_dns_ip` (not `_ips` since it refers to one). We > > > > should also mention in the KIP that only one type of address (ipv4 or > > > ipv6, > > > > based on the first one) will be used - that is the current behaviour > > for > > > > `use_all_dns_ips`. Since we are changing `default` to be exactly the > > > same > > > > as `use_all_dns_ips`, it will be good to mention that explicitly > under > > > > Public Interfaces. > > > > > > > > Regards, > > > > > > > > Rajini > > > > > > > > > > > > On Mon, May 18, 2020 at 1:44 AM Badai Aqrandista > > > > > wrote: > > > > > > > > > Ismael > > > > > > > > > > What do you think of the PR and the explanation regarding the issue > > > > raised > > > > > in KIP-235? > > > > > > > > > > Should I go ahead and build a proper PR? > > > > > > > > > > Thanks > > > > > Badai > > > > > > > > > > On Mon, May 11, 2020 at 8:53 AM Badai Aqrandista < > ba...@confluent.io > > > > > > > > wrote: > > > > > > > > > > > Ismael > > > > > > > > > > > > PR created: https://github.com/apache/kafka/pull/8644/files > > > > > > > > > > > > Also, as this is my first PR, please let me know if I missed > > > anything. > > > > > > > > > > > > Thanks > > > > > > Badai > > > > > > > > > > > > On Mon, May 11, 2020 at 8:19 AM Badai Aqrandista < > > ba...@confluent.io > > > > > > > > > > wrote: > > > > > > > > > > > >> Ismael > > > > > >> > > > > > >> Thank you for responding. > > > > > >> > > > > > >> KIP-235 modified ClientUtils#parseAndValidateAddresses [1] to > > > resolve > > > > an > > > > > >> address alias (i.e. bootstrap server) into multiple addresses. > > This > > > is > > > > > why > > > > > >> it would break SSL hostname verification when the bootstrap > server > > > is > > > > > an IP > > > > > >> address, i.e. it will resolve the IP address to an FQDN and use > > that > > > > > FQDN > > > > > >> in the SSL handshake. > > > > > >> > > > > > >> However, what I am proposing is to modify ClientUtils#resolve > [2], > > > > which > > > > > >> is only used in ClusterConnectionStates#currentAddress [3], to > get > > > the > > > > > >> resolved InetAddress of the address to connect to. And > > > > > >> ClusterConnectionStates#currentAddress is only used by > > > > > >> NetworkClient#initiateConnect [4] to create InetSocketAddress to > > > > > establish > > > > > >> the socket connection to the broker. > > > > > >> > > > > > >> Therefore, as far as I know, this change will not affect higher > > > level > > > > > >> protocol like SSL or SASL. > > > > > >> > > > > > >> PR coming after this. > > > > > >> > > > > > >> Thanks > > > > > >> Badai > > > > > >> > > > > > >> [1] > > > > > >>
[VOTE] KIP-602: Change default value for client.dns.lookup
Hi All, I would like to start the vote on KIP-602: Change default value for client.dns.lookup For reference, here is the KIP wiki: https://cwiki.apache.org/confluence/display/KAFKA/KIP-602%3A+Change+default+value+for+client.dns.lookup And discussion thread: https://lists.apache.org/thread.html/r0e70d3757267c4158f12c05a4e5ac9eb33f2d11ce99d5878b3b4b3f7%40%3Cdev.kafka.apache.org%3E -- Thanks, Badai
Re: [DISCUSS] KIP-602 - Change default value for client.dns.lookup
Ismael What do you think of the PR and the explanation regarding the issue raised in KIP-235? Should I go ahead and build a proper PR? Thanks Badai On Mon, May 11, 2020 at 8:53 AM Badai Aqrandista wrote: > Ismael > > PR created: https://github.com/apache/kafka/pull/8644/files > > Also, as this is my first PR, please let me know if I missed anything. > > Thanks > Badai > > On Mon, May 11, 2020 at 8:19 AM Badai Aqrandista > wrote: > >> Ismael >> >> Thank you for responding. >> >> KIP-235 modified ClientUtils#parseAndValidateAddresses [1] to resolve an >> address alias (i.e. bootstrap server) into multiple addresses. This is why >> it would break SSL hostname verification when the bootstrap server is an IP >> address, i.e. it will resolve the IP address to an FQDN and use that FQDN >> in the SSL handshake. >> >> However, what I am proposing is to modify ClientUtils#resolve [2], which >> is only used in ClusterConnectionStates#currentAddress [3], to get the >> resolved InetAddress of the address to connect to. And >> ClusterConnectionStates#currentAddress is only used by >> NetworkClient#initiateConnect [4] to create InetSocketAddress to establish >> the socket connection to the broker. >> >> Therefore, as far as I know, this change will not affect higher level >> protocol like SSL or SASL. >> >> PR coming after this. >> >> Thanks >> Badai >> >> [1] >> https://github.com/apache/kafka/blob/2.5.0/clients/src/main/java/org/apache/kafka/clients/ClientUtils.java#L51 >> [2] >> https://github.com/apache/kafka/blob/2.5.0/clients/src/main/java/org/apache/kafka/clients/ClientUtils.java#L111 >> [3] >> https://github.com/apache/kafka/blob/2.5.0/clients/src/main/java/org/apache/kafka/clients/ClusterConnectionStates.java#L403 >> [4] >> https://github.com/apache/kafka/blob/2.5.0/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java#L955 >> >> On Sun, May 10, 2020 at 10:18 AM Ismael Juma wrote: >> >>> Hi Badai, >>> >>> I think this is a good change. Can you please address the issues raised >>> by KIP-235? That was the reason why we did not do it previously. >>> >>> Ismael >>> >>> On Mon, Apr 27, 2020 at 5:46 PM Badai Aqrandista >>> wrote: >>> >>>> Hi everyone >>>> >>>> I have opened this KIP to have client.dns.lookup default value changed >>>> to >>>> "use_all_dns_ips". >>>> >>>> >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-602%3A+Change+default+value+for+client.dns.lookup >>>> >>>> Feedback appreciated. >>>> >>>> PS: I'm new here so please let me know if I miss anything. >>>> >>>> -- >>>> Thanks, >>>> Badai >>>> >>> >> >> -- >> Thanks, >> Badai >> >> > > -- > Thanks, > Badai > > -- Thanks, Badai
Re: [DISCUSS] KIP-602 - Change default value for client.dns.lookup
Ismael PR created: https://github.com/apache/kafka/pull/8644/files Also, as this is my first PR, please let me know if I missed anything. Thanks Badai On Mon, May 11, 2020 at 8:19 AM Badai Aqrandista wrote: > Ismael > > Thank you for responding. > > KIP-235 modified ClientUtils#parseAndValidateAddresses [1] to resolve an > address alias (i.e. bootstrap server) into multiple addresses. This is why > it would break SSL hostname verification when the bootstrap server is an IP > address, i.e. it will resolve the IP address to an FQDN and use that FQDN > in the SSL handshake. > > However, what I am proposing is to modify ClientUtils#resolve [2], which > is only used in ClusterConnectionStates#currentAddress [3], to get the > resolved InetAddress of the address to connect to. And > ClusterConnectionStates#currentAddress is only used by > NetworkClient#initiateConnect [4] to create InetSocketAddress to establish > the socket connection to the broker. > > Therefore, as far as I know, this change will not affect higher level > protocol like SSL or SASL. > > PR coming after this. > > Thanks > Badai > > [1] > https://github.com/apache/kafka/blob/2.5.0/clients/src/main/java/org/apache/kafka/clients/ClientUtils.java#L51 > [2] > https://github.com/apache/kafka/blob/2.5.0/clients/src/main/java/org/apache/kafka/clients/ClientUtils.java#L111 > [3] > https://github.com/apache/kafka/blob/2.5.0/clients/src/main/java/org/apache/kafka/clients/ClusterConnectionStates.java#L403 > [4] > https://github.com/apache/kafka/blob/2.5.0/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java#L955 > > On Sun, May 10, 2020 at 10:18 AM Ismael Juma wrote: > >> Hi Badai, >> >> I think this is a good change. Can you please address the issues raised >> by KIP-235? That was the reason why we did not do it previously. >> >> Ismael >> >> On Mon, Apr 27, 2020 at 5:46 PM Badai Aqrandista >> wrote: >> >>> Hi everyone >>> >>> I have opened this KIP to have client.dns.lookup default value changed to >>> "use_all_dns_ips". >>> >>> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-602%3A+Change+default+value+for+client.dns.lookup >>> >>> Feedback appreciated. >>> >>> PS: I'm new here so please let me know if I miss anything. >>> >>> -- >>> Thanks, >>> Badai >>> >> > > -- > Thanks, > Badai > > -- Thanks, Badai
Re: [DISCUSS] KIP-602 - Change default value for client.dns.lookup
Ismael Thank you for responding. KIP-235 modified ClientUtils#parseAndValidateAddresses [1] to resolve an address alias (i.e. bootstrap server) into multiple addresses. This is why it would break SSL hostname verification when the bootstrap server is an IP address, i.e. it will resolve the IP address to an FQDN and use that FQDN in the SSL handshake. However, what I am proposing is to modify ClientUtils#resolve [2], which is only used in ClusterConnectionStates#currentAddress [3], to get the resolved InetAddress of the address to connect to. And ClusterConnectionStates#currentAddress is only used by NetworkClient#initiateConnect [4] to create InetSocketAddress to establish the socket connection to the broker. Therefore, as far as I know, this change will not affect higher level protocol like SSL or SASL. PR coming after this. Thanks Badai [1] https://github.com/apache/kafka/blob/2.5.0/clients/src/main/java/org/apache/kafka/clients/ClientUtils.java#L51 [2] https://github.com/apache/kafka/blob/2.5.0/clients/src/main/java/org/apache/kafka/clients/ClientUtils.java#L111 [3] https://github.com/apache/kafka/blob/2.5.0/clients/src/main/java/org/apache/kafka/clients/ClusterConnectionStates.java#L403 [4] https://github.com/apache/kafka/blob/2.5.0/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java#L955 On Sun, May 10, 2020 at 10:18 AM Ismael Juma wrote: > Hi Badai, > > I think this is a good change. Can you please address the issues raised > by KIP-235? That was the reason why we did not do it previously. > > Ismael > > On Mon, Apr 27, 2020 at 5:46 PM Badai Aqrandista > wrote: > >> Hi everyone >> >> I have opened this KIP to have client.dns.lookup default value changed to >> "use_all_dns_ips". >> >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-602%3A+Change+default+value+for+client.dns.lookup >> >> Feedback appreciated. >> >> PS: I'm new here so please let me know if I miss anything. >> >> -- >> Thanks, >> Badai >> > -- Thanks, Badai
[DISCUSS] KIP-602 - Change default value for client.dns.lookup
Hi everyone I have opened this KIP to have client.dns.lookup default value changed to "use_all_dns_ips". https://cwiki.apache.org/confluence/display/KAFKA/KIP-602%3A+Change+default+value+for+client.dns.lookup Feedback appreciated. PS: I'm new here so please let me know if I miss anything. -- Thanks, Badai
[jira] [Resolved] (KAFKA-9684) Add support for SNI names in SSL request
[ https://issues.apache.org/jira/browse/KAFKA-9684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Badai Aqrandista resolved KAFKA-9684. - Resolution: Not A Problem Java 7 should include SNI names by default. > Add support for SNI names in SSL request > > > Key: KAFKA-9684 > URL: https://issues.apache.org/jira/browse/KAFKA-9684 > Project: Kafka > Issue Type: Improvement > Reporter: Badai Aqrandista >Priority: Minor > > When running Kafka cluster with SSL security behind HA Proxy, we need the > client to send SSL packets with SNI name extension [1]. This will allow HA > Proxy to forward the request to the relevant broker behind it (passthrough). > Java 7 and higher supports this by adding SNIHostName [2] to SSLParameters > [3]. > [1] https://www.ietf.org/rfc/rfc6066.txt > [2] https://docs.oracle.com/javase/8/docs/api/javax/net/ssl/SNIHostName.html > [3] > https://docs.oracle.com/javase/8/docs/api/javax/net/ssl/SSLParameters.html#setServerNames-java.util.List- -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9684) Add support for SNI names in SSL request
Badai Aqrandista created KAFKA-9684: --- Summary: Add support for SNI names in SSL request Key: KAFKA-9684 URL: https://issues.apache.org/jira/browse/KAFKA-9684 Project: Kafka Issue Type: Improvement Reporter: Badai Aqrandista When running Kafka cluster with SSL security behind HA Proxy, we need the client to send SSL packets with SNI name extension [1]. This will allow HA Proxy to forward the request to the relevant broker behind it (passthrough). Java 7 and higher supports this by adding SNIHostName [2] to SSLParameters [3]. [1] https://www.ietf.org/rfc/rfc6066.txt [2] https://docs.oracle.com/javase/8/docs/api/javax/net/ssl/SNIHostName.html [3] https://docs.oracle.com/javase/8/docs/api/javax/net/ssl/SSLParameters.html#setServerNames-java.util.List- -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9540) Application getting "Could not find the standby task 0_4 while closing it" error
Badai Aqrandista created KAFKA-9540:
---
Summary: Application getting "Could not find the standby task 0_4
while closing it" error
Key: KAFKA-9540
URL: https://issues.apache.org/jira/browse/KAFKA-9540
Project: Kafka
Issue Type: Bug
Components: streams
Reporter: Badai Aqrandista
Because of this the following line, there is a possibility that some standby
tasks might not be created:
https://github.com/apache/kafka/blob/2.4.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java#L436
Then causing this line to not adding the task to standby task list:
https://github.com/apache/kafka/blob/2.4.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java#L299
But this line assumes that all standby tasks are to be created and add it to
the standby list:
https://github.com/apache/kafka/blob/2.4.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java#L168
This results in user getting this error message on the next
PARTITION_ASSIGNMENT state:
{noformat}
Could not find the standby task 0_4 while closing it
(org.apache.kafka.streams.processor.internals.AssignedStandbyTasks:74)
{noformat}
But the harm caused by this issue is minimal: No standby task for some
partitions. And it is recreated on the next rebalance anyway. So, I suggest
lowering this message to WARN. Or probably check to WARN when standby task
could not be created.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-9459) MM2 sync topic config does work
Badai Aqrandista created KAFKA-9459:
---
Summary: MM2 sync topic config does work
Key: KAFKA-9459
URL: https://issues.apache.org/jira/browse/KAFKA-9459
Project: Kafka
Issue Type: Bug
Components: mirrormaker
Affects Versions: 2.4.0
Reporter: Badai Aqrandista
I have MM2 configured as follow:
{code:java}
{
"name": "mm2-from-1-to-2",
"config": {
"connector.class":"org.apache.kafka.connect.mirror.MirrorSourceConnector",
"topics":"foo",
"key.converter":
"org.apache.kafka.connect.converters.ByteArrayConverter",
"value.converter":
"org.apache.kafka.connect.converters.ByteArrayConverter",
"sync.topic.configs.enabled":"true",
"sync.topic.configs.interval.seconds": 60,
"sync.topic.acls.enabled": "false",
"replication.factor": 1,
"offset-syncs.topic.replication.factor": 1,
"heartbeats.topic.replication.factor": 1,
"checkpoints.topic.replication.factor": 1,
"target.cluster.alias":"dest",
"target.cluster.bootstrap.servers":"dest.example.com:9092",
"source.cluster.alias":"src",
"source.cluster.bootstrap.servers":"src.example.com:9092",
"tasks.max": 1}
}
{code}
Topic "foo" is configured with "cleanup.policy=compact". But after waiting for
15 minutes, I still don't see "src.foo" in the destination cluster has
"cleanup.policy=compact".
I had the connect node to run in TRACE level and I could not find any calls to
describeConfigs
(https://github.com/apache/kafka/blob/2.4.0/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L327).
This implies it never actually get a list of topics that it needs to get topic
configs from.
And I am suspecting this code
(https://github.com/apache/kafka/blob/2.4.0/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L214-L220):
{code:java}
private Set topicsBeingReplicated() {
return knownTopicPartitions.stream()
.map(x -> x.topic())
.distinct()
.filter(x -> knownTargetTopics.contains(formatRemoteTopic(x)))
.collect(Collectors.toSet());
}
{code}
knownTopicPartitions contains topic-partitions from the source cluster.
knownTargetTopics contains topic-partitions from the target cluster, whose
topic names contain source alias already.
So, why is topicsBeingReplicated (list of topic-partitions from source cluster)
being filtered using knownTopicPartitions (list of topic-partitions from target
cluster)?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-9275) Print assignment and IP address in the log message when consumer leaves/removed from the group
Badai Aqrandista created KAFKA-9275: --- Summary: Print assignment and IP address in the log message when consumer leaves/removed from the group Key: KAFKA-9275 URL: https://issues.apache.org/jira/browse/KAFKA-9275 Project: Kafka Issue Type: Improvement Reporter: Badai Aqrandista In order to simplify identification of which member is causing rebalance, can we add the IP address and the list of topic-partitions assigned to the member that leaves/removed from the group? And especially in the "reason" string for rebalance: https://github.com/apache/kafka/blob/2.4/core/src/main/scala/kafka/coordinator/group/GroupCoordinator.scala#L964 https://github.com/apache/kafka/blob/2.4/core/src/main/scala/kafka/coordinator/group/GroupCoordinator.scala#L972 https://github.com/apache/kafka/blob/2.4/core/src/main/scala/kafka/coordinator/group/GroupCoordinator.scala#L1144 https://github.com/apache/kafka/blob/2.4/core/src/main/scala/kafka/coordinator/group/GroupCoordinator.scala#L493 This will allow a much faster investigation when a consumer group is stuck in rebalancing state. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-8562) SASL_SSL still performs reverse DNS lookup despite KAFKA-5051
Badai Aqrandista created KAFKA-8562:
---
Summary: SASL_SSL still performs reverse DNS lookup despite
KAFKA-5051
Key: KAFKA-8562
URL: https://issues.apache.org/jira/browse/KAFKA-8562
Project: Kafka
Issue Type: Bug
Reporter: Badai Aqrandista
When using SASL_SSL, the Kafka client performs a reverse DNS lookup to resolve
IP to DNS. So, this circumvent the security fix made in KAFKA-5051.
This is the line of code from AK 2.2 where it performs the lookup:
https://github.com/apache/kafka/blob/2.2.0/clients/src/main/java/org/apache/kafka/common/network/SaslChannelBuilder.java#L205
Following log messages show that consumer initially tried to connect with IP
address 10.0.2.15. Then suddenly it created SaslClient with a hostname:
{code:java}
[2019-06-18 06:23:36,486] INFO Kafka commitId: 00d486623990ed9d
(org.apache.kafka.common.utils.AppInfoParser)
[2019-06-18 06:23:36,487] DEBUG [Consumer clientId=KafkaStore-reader-_schemas,
groupId=schema-registry-10.0.2.15-18081] Kafka consumer initialized
(org.apache.kafka.clients.consumer.KafkaConsumer)
[2019-06-18 06:23:36,505] DEBUG [Consumer clientId=KafkaStore-reader-_schemas,
groupId=schema-registry-10.0.2.15-18081] Initiating connection to node
10.0.2.15:19094 (id: -1 rack: null) using address /10.0.2.15
(org.apache.kafka.clients.NetworkClient)
[2019-06-18 06:23:36,512] DEBUG Set SASL client state to
SEND_APIVERSIONS_REQUEST
(org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2019-06-18 06:23:36,515] DEBUG Creating SaslClient:
client=null;service=kafka;serviceHostname=quickstart.confluent.io;mechs=[PLAIN]
(org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
{code}
Thanks
Badai
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-8546) Call System#runFinalization to avoid memory leak caused by JDK-6293787
Badai Aqrandista created KAFKA-8546:
---
Summary: Call System#runFinalization to avoid memory leak caused
by JDK-6293787
Key: KAFKA-8546
URL: https://issues.apache.org/jira/browse/KAFKA-8546
Project: Kafka
Issue Type: Bug
Affects Versions: 2.0.1
Reporter: Badai Aqrandista
When a heavily used broker uses gzip compression on all topics, sometime you
can hit GC pauses greater than zookeeper.session.timeout.ms of 6000ms. This is
caused by memory leak caused by JDK-6293787
([https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6293787]), which is
caused by JDK-4797189
([https://bugs.java.com/bugdatabase/view_bug.do?bug_id=4797189]).
In summary, this is what happen:
* Inflater class contains finalizer method.
* Whenever a class with finalizer method is instantiated, a Finalizer object
is created.
* GC finalizer thread is responsible to process all Finalizer objects.
* If the rate of Finalizer object creation exceed the rate of GC finalizer
thread ability to process it, Finalizer object number grows continuously, and
eventually triggers full GC (because it is stored in Old Gen).
Following stack trace shows what happen when a process is frozen doing full GC:
{code:java}
kafka-request-handler-13 Runnable Thread ID: 79
java.util.zip.Inflater.inflateBytes(long, byte[], int, int) Inflater.java
java.util.zip.Inflater.inflate(byte[], int, int) Inflater.java:259
java.util.zip.InflaterInputStream.read(byte[], int, int)
InflaterInputStream.java:152
java.util.zip.GZIPInputStream.read(byte[], int, int) GZIPInputStream.java:117
java.io.BufferedInputStream.fill() BufferedInputStream.java:246
java.io.BufferedInputStream.read() BufferedInputStream.java:265
java.io.DataInputStream.readByte() DataInputStream.java:265
org.apache.kafka.common.utils.ByteUtils.readVarint(DataInput)
ByteUtils.java:168
org.apache.kafka.common.record.DefaultRecord.readFrom(DataInput, long, long,
int, Long) DefaultRecord.java:292
org.apache.kafka.common.record.DefaultRecordBatch$1.readNext(long, long, int,
Long) DefaultRecordBatch.java:264
org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next()
DefaultRecordBatch.java:563
org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next()
DefaultRecordBatch.java:532
org.apache.kafka.common.record.DefaultRecordBatch.iterator()
DefaultRecordBatch.java:327
scala.collection.convert.Wrappers$JIterableWrapper.iterator()
Wrappers.scala:54
scala.collection.IterableLike$class.foreach(IterableLike, Function1)
IterableLike.scala:72
scala.collection.AbstractIterable.foreach(Function1) Iterable.scala:54
kafka.log.LogValidator$$anonfun$validateMessagesAndAssignOffsetsCompressed$1.apply(MutableRecordBatch)
LogValidator.scala:267
kafka.log.LogValidator$$anonfun$validateMessagesAndAssignOffsetsCompressed$1.apply(Object)
LogValidator.scala:259
scala.collection.Iterator$class.foreach(Iterator, Function1)
Iterator.scala:891
scala.collection.AbstractIterator.foreach(Function1) Iterator.scala:1334
scala.collection.IterableLike$class.foreach(IterableLike, Function1)
IterableLike.scala:72
scala.collection.AbstractIterable.foreach(Function1) Iterable.scala:54
kafka.log.LogValidator$.validateMessagesAndAssignOffsetsCompressed(MemoryRecords,
LongRef, Time, long, CompressionCodec, CompressionCodec, boolean, byte,
TimestampType, long, int, boolean) LogValidator.scala:259
kafka.log.LogValidator$.validateMessagesAndAssignOffsets(MemoryRecords,
LongRef, Time, long, CompressionCodec, CompressionCodec, boolean, byte,
TimestampType, long, int, boolean) LogValidator.scala:70
kafka.log.Log$$anonfun$append$2.liftedTree1$1(LogAppendInfo, ObjectRef,
LongRef, long) Log.scala:771
kafka.log.Log$$anonfun$append$2.apply() Log.scala:770
kafka.log.Log$$anonfun$append$2.apply() Log.scala:752
kafka.log.Log.maybeHandleIOException(Function0, Function0) Log.scala:1842
kafka.log.Log.append(MemoryRecords, boolean, boolean, int) Log.scala:752
kafka.log.Log.appendAsLeader(MemoryRecords, int, boolean) Log.scala:722
kafka.cluster.Partition$$anonfun$13.apply() Partition.scala:660
kafka.cluster.Partition$$anonfun$13.apply() Partition.scala:648
kafka.utils.CoreUtils$.inLock(Lock, Function0) CoreUtils.scala:251
kafka.utils.CoreUtils$.inReadLock(ReadWriteLock, Function0)
CoreUtils.scala:257
kafka.cluster.Partition.appendRecordsToLeader(MemoryRecords, boolean, int)
Partition.scala:647
kafka.server.ReplicaManager$$anonfun$appendToLocalLog$2.apply(Tuple2)
ReplicaManager.scala:745
kafka.server.ReplicaManager$$anonfun$appendToLocalLog$2.apply(Object)
ReplicaManager.scala:733
scala.collection.TraversableLike$$anonfun$map$1.apply(Object)
TraversableLike.scala:234
scala.collection.TraversableLike$$anonfun$map$1.apply(Object)
TraversableLike.scala:234
scala.collection.mutable.HashMap$$anonfun$foreach$1
[jira] [Created] (KAFKA-8501) Remove key and value from exception message
Badai Aqrandista created KAFKA-8501: --- Summary: Remove key and value from exception message Key: KAFKA-8501 URL: https://issues.apache.org/jira/browse/KAFKA-8501 Project: Kafka Issue Type: Bug Reporter: Badai Aqrandista KAFKA-7510 moves the WARN messages that contain key and value to TRACE level. But the exceptions still contain key and value. These are the two in RecordCollectorImpl: [https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/processor/internals/RecordCollectorImpl.java#L185-L196] [https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/processor/internals/RecordCollectorImpl.java#L243-L254] Can these be modified as well to remove key and value from the error message, which we don't know what log level it will be printed in? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7760) Add broker configuration to set minimum value for segment.bytes and segment.ms
Badai Aqrandista created KAFKA-7760: --- Summary: Add broker configuration to set minimum value for segment.bytes and segment.ms Key: KAFKA-7760 URL: https://issues.apache.org/jira/browse/KAFKA-7760 Project: Kafka Issue Type: Improvement Reporter: Badai Aqrandista If someone set segment.bytes or segment.ms at topic level to a very small value (e.g. segment.bytes=1000 or segment.ms=1000), Kafka will generate a very high number of segment files. This can bring down the whole broker due to hitting the maximum open file (for log) or maximum number of mmap-ed file (for index). To prevent that from happening, I would like to suggest adding two new items to the broker configuration: * min.topic.segment.bytes, defaults to 1048576: The minimum value for segment.bytes. When someone sets topic configuration segment.bytes to a value lower than this, Kafka throws an error INVALID VALUE. * min.topic.segment.ms, defaults to 360: The minimum value for segment.ms. When someone sets topic configuration segment.ms to a value lower than this, Kafka throws an error INVALID VALUE. Thanks Badai -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7754) zookeeper-security-migration.sh sets the root ZNode as world-readable
Badai Aqrandista created KAFKA-7754:
---
Summary: zookeeper-security-migration.sh sets the root ZNode as
world-readable
Key: KAFKA-7754
URL: https://issues.apache.org/jira/browse/KAFKA-7754
Project: Kafka
Issue Type: Bug
Components: security
Affects Versions: 2.0.1
Reporter: Badai Aqrandista
If I start broker with {{zookeeper.set.acl=true}} from the first time I start
the broker, the root ZNode is not set to be world-readable to allow other
application to share the Zookeeper ensemble with chroot.
But if I run {{zookeeper-security-migration.sh}} with {{–zookeeper.acl
secure}}, the root ZNode becomes world-readable. Is this correct?
{noformat}
root@localhost:/# zookeeper-shell localhost:2181
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is enabled
[zk: localhost:2181(CONNECTING) 0]
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
WATCHER::
WatchedEvent state:SaslAuthenticated type:None path:null
[zk: localhost:2181(CONNECTED) 0] getAcl /
'world,'anyone
: cdrwa
[zk: localhost:2181(CONNECTED) 1] getAcl /brokers
'world,'anyone
: r
'sasl,'kafkabroker
: cdrwa
[zk: localhost:2181(CONNECTED) 2] quit
Quitting...
root@localhost:/# zookeeper-security-migration --zookeeper.acl secure
--zookeeper.connect localhost:2181
root@localhost:/# zookeeper-shell localhost:2181
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is enabled
[zk: localhost:2181(CONNECTING) 0]
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
WATCHER::
WatchedEvent state:SaslAuthenticated type:None path:null
[zk: localhost:2181(CONNECTED) 0] getAcl /
'world,'anyone
: r
'sasl,'kafkabroker
: cdrwa
[zk: localhost:2181(CONNECTED) 1] getAcl /brokers
'world,'anyone
: r
'sasl,'kafkabroker
: cdrwa
[zk: localhost:2181(CONNECTED) 2]
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-7467) NoSuchElementException is raised because controlBatch is empty
Badai Aqrandista created KAFKA-7467:
---
Summary: NoSuchElementException is raised because controlBatch is
empty
Key: KAFKA-7467
URL: https://issues.apache.org/jira/browse/KAFKA-7467
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 1.1.0
Reporter: Badai Aqrandista
Somehow, log cleaner died because of NoSuchElementException when it calls
onControlBatchRead:
{noformat}
[2018-09-25 14:18:31,088] INFO Cleaner 0: Cleaning segment 0 in log
__consumer_offsets-45 (largest timestamp Fri Apr 27 16:12:39 CDT 2018) into 0,
discarding deletes. (kafka.log.LogCleaner)
[2018-09-25 14:18:31,092] ERROR [kafka-log-cleaner-thread-0]: Error due to
(kafka.log.LogCleaner)
java.util.NoSuchElementException
at java.util.Collections$EmptyIterator.next(Collections.java:4189)
at
kafka.log.CleanedTransactionMetadata.onControlBatchRead(LogCleaner.scala:945)
at
kafka.log.Cleaner.kafka$log$Cleaner$$shouldDiscardBatch(LogCleaner.scala:636)
at kafka.log.Cleaner$$anon$5.checkBatchRetention(LogCleaner.scala:573)
at
org.apache.kafka.common.record.MemoryRecords.filterTo(MemoryRecords.java:157)
at
org.apache.kafka.common.record.MemoryRecords.filterTo(MemoryRecords.java:138)
at kafka.log.Cleaner.cleanInto(LogCleaner.scala:604)
at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:518)
at kafka.log.Cleaner$$anonfun$doClean$4.apply(LogCleaner.scala:462)
at kafka.log.Cleaner$$anonfun$doClean$4.apply(LogCleaner.scala:461)
at scala.collection.immutable.List.foreach(List.scala:392)
at kafka.log.Cleaner.doClean(LogCleaner.scala:461)
at kafka.log.Cleaner.clean(LogCleaner.scala:438)
at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:305)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:291)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
[2018-09-25 14:18:31,093] INFO [kafka-log-cleaner-thread-0]: Stopped
(kafka.log.LogCleaner)
{noformat}
The following code does not seem to expect the controlBatch to be empty:
https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/LogCleaner.scala#L946
{noformat}
def onControlBatchRead(controlBatch: RecordBatch): Boolean = {
consumeAbortedTxnsUpTo(controlBatch.lastOffset)
val controlRecord = controlBatch.iterator.next()
val controlType = ControlRecordType.parse(controlRecord.key)
val producerId = controlBatch.producerId
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-7379) send.buffer.bytes should be allowed to set -1 in KafkaStreams
Badai Aqrandista created KAFKA-7379: --- Summary: send.buffer.bytes should be allowed to set -1 in KafkaStreams Key: KAFKA-7379 URL: https://issues.apache.org/jira/browse/KAFKA-7379 Project: Kafka Issue Type: Bug Components: streams Affects Versions: 2.0.0 Reporter: Badai Aqrandista send.buffer.bytes and receive.buffer.bytes are declared with atLeast(0) constraint in StreamsConfig, whereas -1 should be also allowed to set. This is like KAFKA-6891. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-6939) Change the default of log.message.timestamp.difference.max.ms to 500 years
Badai Aqrandista created KAFKA-6939: --- Summary: Change the default of log.message.timestamp.difference.max.ms to 500 years Key: KAFKA-6939 URL: https://issues.apache.org/jira/browse/KAFKA-6939 Project: Kafka Issue Type: Improvement Reporter: Badai Aqrandista If producer incorrectly provides timestamp in microsecond (not in millisecond), the record is accepted by default and can cause broker to roll the segment files continuously. And on a heavily used broker, this will generate a lot of index files, which then causes the broker to hit `vm.max_map_count`. So I'd like to suggest changing the default for log.message.timestamp.difference.max.ms to 157680 (500 years * 365 days * 86400 seconds * 1000). This would reject timestamp in microsecond from producer and still allow most historical data to be stored in Kafka. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-6756) client.id and group.id validation in the old vs new consumer
Badai Aqrandista created KAFKA-6756: --- Summary: client.id and group.id validation in the old vs new consumer Key: KAFKA-6756 URL: https://issues.apache.org/jira/browse/KAFKA-6756 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 1.1.0, 0.10.2.1 Reporter: Badai Aqrandista It looks like the old consumer that is based on the Scala code validates "client.id" and "group.id" using this code: [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/common/Config.scala#L25] [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/consumer/ConsumerConfig.scala#L60] However, the new consumer uses the Java code that does not validate "client.id" and "group.id" at all here: [https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java#L264] [https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java#L298] So the new consumer does not validate "client.id" and "group.id" like the old consumer. Either way, the documentation never specify the valid character for "client.id" and "group.id". Is this a bug or by design? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
