Build failed in Jenkins: kafka-trunk-jdk11 #902

See Changes: [matthias] MINOR: don't require key serde in join materialized (#7557) [matthias] MINOR: KIP-307 upgrade guide docs (#7547) [matthias] MINOR:Upgrade guide updates for KIP-479 (#7550)

Re: [DISCUSS] KIP-399: Extend ProductionExceptionHandler to cover serialization exceptions

Some more details: Changelog Topics: in the DSL, serialization happens in the store level (MeteredXxxStore) and hence any serialization exception should be thrown to the `Processor` before the message is written to the changelog topic or underlying store. When the actually write to the changelog t

Re: [VOTE] KIP-537: Increase default zookeeper session timeout

+1 (binding) On Mon, Oct 21, 2019, 5:28 PM Jason Gustafson wrote: > I'd like to start a vote for KIP-537: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-537%3A+Increase+default+zookeeper+session+timeout > . > > +1 from me > > Thanks, > Jason >

Re: [DISCUSS] KIP-535: Allow state stores to serve stale reads during rebalance

Some more follow up thoughts: 11) If we get a store handle of an active(restoring) task, and the task transits to running, does the store handle become invalid and a new one must be retrieved? Or can we "switch it out" underneath -- for this case, how does the user know when they start to query th

Jenkins build is back to normal : kafka-trunk-jdk11 #901

See

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

It will be very helpful in having the feature branch in apache/kafka repo instead of having it in private repos. I plan to close the current PR and create a new PR from apache/kafka/tiered-storage branch when it is ready. Thanks, Satish. On Mon, Oct 21, 2019 at 9:16 PM Harsha wrote: > Hi All,

Build failed in Jenkins: kafka-2.4-jdk8 #31

See Changes: [matthias] MINOR: don't require key serde in join materialized (#7557) [matthias] MINOR: KIP-307 upgrade guide docs (#7547) -- [...truncated 2.93 MB...] org.ap

Re: [VOTE] KIP-537: Increase default zookeeper session timeout

+1(non-binding). This is such an easy default quality of life change On Mon, Oct 21, 2019, 7:28 PM Jason Gustafson wrote: > I'd like to start a vote for KIP-537: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-537%3A+Increase+default+zookeeper+session+timeout > . > > +1 from me > > Tha

Re: [DISCUSS] KIP-535: Allow state stores to serve stale reads during rebalance

Thanks for the KIP. Couple of comments: 1) With regard to KIP-441, my current understanding is that the lag information is only reported to the leader (please correct me if I am wrong). This seems to be quite a limitation to actually use the lag information. 2) The idea of the metadata API is ac

Re: [DISCUSS] KIP-535: Allow state stores to serve stale reads during rebalance

>>I'm wondering, rather than putting "acceptable lag" into the configuration at all, or even making it a parameter on `allMetadataForKey`, why not just _always_ return all available metadata (including active/standby or lag) and let the caller decide to which node they want to route the query? +1 o

[VOTE] KIP-537: Increase default zookeeper session timeout

I'd like to start a vote for KIP-537: https://cwiki.apache.org/confluence/display/KAFKA/KIP-537%3A+Increase+default+zookeeper+session+timeout . +1 from me Thanks, Jason

[jira] [Reopened] (KAFKA-9075) Extend documentation for usage of GlobalKTable vs KTable

[ https://issues.apache.org/jira/browse/KAFKA-9075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax reopened KAFKA-9075: I think it would actually be good to extend the documentation and explain the semantic differe

Re: [VOTE] KIP-538: Add a metric tracking the number of open connections with a given SSL cipher type

+1 Thanks for leading this :) On Mon, Oct 21, 2019 at 3:43 PM Colin McCabe wrote: > > Hi all, > > I'd like to start the vote on KIP-538: Add a metric tracking the number of > open connections with a given SSL cipher type. > > KIP: https://cwiki.apache.org/confluence/x/txPjBw > > Discussion thre

Re: [DISCUSS] KIP-541: Create a fetch.max.bytes configuration for the broker

Thanks Collin. I read this statement " Fetch request from replicas will also be affected by the *fetch.max.bytes* limit." Which made me think whether this was referring to replica fetcher byte size. But thanks for clarifying. Regards, On Tue, 22 Oct 2019 at 00:26, Colin McCabe wrote: > On Mon

Re: [DISCUSS] KIP-541: Create a fetch.max.bytes configuration for the broker

On Mon, Oct 21, 2019, at 15:52, M. Manna wrote: > Hello Colin, > > The KIP looks concise. My comments are below. > > replica.fetch.max.bytes is relevant when there is replication involved, so > I am trying to understand how fetch.max.bytes for a broker will play a role > here. Apologies for any l

Re: [DISCUSS] KIP-409: Allow creating under-replicated topics and partitions

Hi Mickael, We don't have any official way for brokers to join the cluster other than showing up and registering themselves in ZK. Similarly, we don't have any way of removing brokers from the cluster other than simply removing them and removing their znodes from ZooKeeper. If we wanted to ch

Re: [DISCUSS] KIP-541: Create a fetch.max.bytes configuration for the broker

Hello Colin, The KIP looks concise. My comments are below. replica.fetch.max.bytes is relevant when there is replication involved, so I am trying to understand how fetch.max.bytes for a broker will play a role here. Apologies for any limited assumptions (always trying to catchup with Kafka :). A

[jira] [Resolved] (KAFKA-9075) Create FAQ documentation for usage of GlobalKTable vs KTable

[ https://issues.apache.org/jira/browse/KAFKA-9075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Chen resolved KAFKA-9075. Resolution: Not A Problem > Create FAQ documentation for usage of GlobalKTable vs KTable >

[VOTE] KIP-538: Add a metric tracking the number of open connections with a given SSL cipher type

Hi all, I'd like to start the vote on KIP-538: Add a metric tracking the number of open connections with a given SSL cipher type. KIP: https://cwiki.apache.org/confluence/x/txPjBw Discussion thread: https://lists.apache.org/thread.html/b1ee845ef3a3d4fe4d83da81d5444cf00e43840d3b2710487255e6eb@%

[DISCUSS] KIP-541: Create a fetch.max.bytes configuration for the broker

Hi all, I wrote a KIP about creating a fetch.max.bytes configuration for the broker. Please take a look here: https://cwiki.apache.org/confluence/x/4g73Bw thanks, Colin

Re: [DISCUSS] KIP-535: Allow state stores to serve stale reads during rebalance

Hi Navinder, Thanks for the KIP, I have a high level question about the proposed API regarding: "StreamsMetadataState::allMetadataForKey(boolean enableReplicaServing...)" I'm wondering if it's more general to refactor it to "allMetadataForKey(long tolerableDataStaleness, ...)", and when it's set

Build failed in Jenkins: kafka-trunk-jdk8 #3987

See Changes: [github] MINOR: system test clean up (#7552) -- [...truncated 2.70 MB...] org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED org.apac

Re: [DISCUSS] KIP-533: Add default api timeout to AdminClient

Thanks for the KIP, this makes sense to me. It will be good to align the AdminClient better with the consumer and producer when it comes to timeouts. Ismael On Wed, Oct 9, 2019 at 12:06 PM Jason Gustafson wrote: > Hi All, > > I wrote a short KIP to address a longstanding issue with timeout beha

[DISCUSS] KIP-280: Enhanced log compaction

Hi All, We are bring back the KIP-280 to live with small correct for the discussion & voting. Thanks to previous author Luis Cabral on the KIP-280 initiation and we are taking over to complete and get it into 2.4... Below is the correction that we made to the existing KIP-280: * Allowing t

[jira] [Resolved] (KAFKA-8940) Flaky Test SmokeTestDriverIntegrationTest.shouldWorkWithRebalance

[ https://issues.apache.org/jira/browse/KAFKA-8940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-8940. -- Fix Version/s: 2.5.0 Resolution: Fixed > Flaky Test SmokeTestDriverIntegrationTest.shoul

Re: [DISCUSS] KIP-535: Allow state stores to serve stale reads during rebalance

Hey Navinder, Thanks for the KIP! I've been reading over the discussion thus far, and I have a couple of thoughts to pile on as well: It seems confusing to propose the API in terms of the current system state, but also propose how the API would look if/when KIP-441 is implemented. It occurs to me

Re: [DISCUSS] KIP-538: Add a metric tracking the number of open connections with a given SSL cipher type

Sounds good to me. On Wed, Oct 16, 2019 at 11:58 AM Colin McCabe wrote: > > Hi all, > > I wrote a short KIP about adding a metric to track the number of open > connections with a given SSL cipher type. Take a look here: > https://cwiki.apache.org/confluence/x/txPjBw > > best, > Colin

[jira] [Created] (KAFKA-9075) Create FAQ documentation for usage of GlobalKTable vs KTable

Boyang Chen created KAFKA-9075: -- Summary: Create FAQ documentation for usage of GlobalKTable vs KTable Key: KAFKA-9075 URL: https://issues.apache.org/jira/browse/KAFKA-9075 Project: Kafka Issue

Re: [DISCUSS] KIP-533: Add default api timeout to AdminClient

Hi Jason, Thanks for the KIP. I think this is a long-overdue improvement. It would be good to clarify that the "timeout" values for the options classes will now refer to the API timeouts, not to the request timeouts. I think this is implied by the KIP, but it would be good to spell it out. O

Build failed in Jenkins: kafka-trunk-jdk11 #900

See Changes: [github] MINOR: system test clean up (#7552) -- [...truncated 2.71 MB...] org.apache.kafka.streams.TopologyTestDriverTest > shouldCloseProcessor[Eos enable

Re: [DISCUSS] KIP-538: Add a metric tracking the number of open connections with a given SSL cipher type

+1 Looks reasonable to me. Tracking this per-processor is probably not that useful, but I agree it simplifies the implementation. -Jason On Wed, Oct 16, 2019 at 11:58 AM Colin McCabe wrote: > Hi all, > > I wrote a short KIP about adding a metric to track the number of open > connections with a

Build failed in Jenkins: kafka-trunk-jdk8 #3986

See Changes: [rajinisivaram] KAFKA-8943: Move SecurityProviderCreator to -- [...truncated 7.60 MB...] org.apache.kafka.streams.integration.EosIntegrationTest > shouldBe

[jira] [Created] (KAFKA-9074) Connect's Values class does not parse time or timestamp values from string literals

Randall Hauch created KAFKA-9074: Summary: Connect's Values class does not parse time or timestamp values from string literals Key: KAFKA-9074 URL: https://issues.apache.org/jira/browse/KAFKA-9074 Pro

Re: [DISCUSS] KIP-506: Allow setting SCRAM password via Admin interface

On Mon, Oct 21, 2019, at 03:59, Viktor Somogyi-Vass wrote: > Hey Tom, > > Bringing over the conversation from KIP-500 and also adding my > observations. Let me capture them in points. > > 1. Perhaps I'm a bit aggressive in introducing new features but I'm not > sure if we need sasl.scram.password

Re: [DISCUSS] KIP-514: Add a bounded flush() API to Kafka Producer

Hi Radai, It seems reasonable to me. best, Colin On Mon, Oct 21, 2019, at 09:52, radai wrote: > yet another bump. > > can we please have a vote if there are no objections ? > > On Wed, Sep 25, 2019 at 1:28 PM radai wrote: > > > > bump. > > > > so if no more concerns, can we move to a vote on

Re: [DISCUSS] KIP-518: Allow listing consumer groups per state

Bump Now that the rush for 2.4.0 is ending I hope to get some feedback Thanks On Mon, Sep 9, 2019 at 5:44 PM Mickael Maison wrote: > > Hi, > > I have created a KIP to allow listing groups per state: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-518%3A+Allow+listing+consumer+groups+per+

Re: [DISCUSS] KIP-514: Add a bounded flush() API to Kafka Producer

yet another bump. can we please have a vote if there are no objections ? On Wed, Sep 25, 2019 at 1:28 PM radai wrote: > > bump. > > so if no more concerns, can we move to a vote on this ? > > On Fri, Sep 13, 2019 at 10:05 AM radai wrote: > > > > we have a lot of processes that need a time-bound

Re: [DISCUSS] KIP-409: Allow creating under-replicated topics and partitions

Thanks Stanislav and Colin for the feedback. I've updated the KIP to make it simpler. It's not updating the CreateTopics/CreatePartitions RPCs anymore. I've kept the broker setting so admins can keep the current behaviour but simplified it to be either enabled or disabled. I've also kept the obse

[jira] [Created] (KAFKA-9073) Kafka Streams State stuck in rebalancing after one of the StreamThread encounters IllegalStateException

amuthan Ganeshan created KAFKA-9073: --- Summary: Kafka Streams State stuck in rebalancing after one of the StreamThread encounters IllegalStateException Key: KAFKA-9073 URL: https://issues.apache.org/jira/browse/K

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

Nice, I was just wondering about this! I'll make sure to look at the PR and maybe try it out on our infra. Do you want to initiate a vote too? Thanks, Viktor On Mon, Oct 21, 2019 at 5:46 PM Harsha wrote: > Hi All, > Thanks for the initial feedback on the KIP-405. We opened a PR > he

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

Hi All, Thanks for the initial feedback on the KIP-405. We opened a PR here https://github.com/apache/kafka/pull/7561 . Please take a look and let us know if you have any questions. Since this feature is being developed by engineers from different companies we would like to open a feat

Build failed in Jenkins: kafka-2.4-jdk8 #30

See Changes: [rajinisivaram] KAFKA-8943: Move SecurityProviderCreator to -- [...truncated 5.09 MB...] kafka.server.MetadataRequestTest > testNoTopicsRequest STARTED kafka.se

[jira] [Created] (KAFKA-9072) Add Section to Streams Developer Guide for Topology Naming (KIP-307)

Bill Bejeck created KAFKA-9072: -- Summary: Add Section to Streams Developer Guide for Topology Naming (KIP-307) Key: KAFKA-9072 URL: https://issues.apache.org/jira/browse/KAFKA-9072 Project: Kafka

[jira] [Resolved] (KAFKA-8081) Flaky Test TopicCommandWithAdminClientTest#testDescribeUnderMinIsrPartitions

[ https://issues.apache.org/jira/browse/KAFKA-8081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viktor Somogyi-Vass resolved KAFKA-8081. Assignee: Viktor Somogyi-Vass Resolution: Fixed > Flaky Test TopicCommandWith

Re: [DISCUSS] KIP-506: Allow setting SCRAM password via Admin interface

Hey Tom, Bringing over the conversation from KIP-500 and also adding my observations. Let me capture them in points. 1. Perhaps I'm a bit aggressive in introducing new features but I'm not sure if we need sasl.scram.password.change.enabled. As I understand it controls whether your can use the Adm

[jira] [Resolved] (KAFKA-8964) Refactor Stream-Thread-level Metrics

[ https://issues.apache.org/jira/browse/KAFKA-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Cadonna resolved KAFKA-8964. -- Resolution: Fixed > Refactor Stream-Thread-level Metrics > ---

[jira] [Resolved] (KAFKA-8943) Move SecurityProviderCreator to a public package

[ https://issues.apache.org/jira/browse/KAFKA-8943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram resolved KAFKA-8943. --- Assignee: Manikumar Resolution: Fixed > Move SecurityProviderCreator to a public packag

[jira] [Created] (KAFKA-9071) transactional.id.expiration.ms config value should be implemented as a Long

Joost van de Wijgerd created KAFKA-9071: --- Summary: transactional.id.expiration.ms config value should be implemented as a Long Key: KAFKA-9071 URL: https://issues.apache.org/jira/browse/KAFKA-9071