Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1846

See

Re: Query regarding implementation of KStreams with Hbase

Kafka Streams is designed to read and write from a broker cluster. It's not designed to write data to different system like HBase. If you want to get data from Kafka to HBase, you should use Kafka Connect. Of course, it's possible (but not recommended) to implement your own `Processor` and do

Query regarding implementation of KStreams with Hbase

Hi team, There is lot in internet where Kstreams read data from topic , perform some transformation and write it back to a topic. But I wonder if write to hbase table is possible with KStream?. And what I mean by this is that we should read data from topic using KStream , perform some operations

Re: [DISCUSS] KIP-931: Flag to ignore unused message attribute field

Hi Kirk, Yes, the pressure in broker comes from the message format down conversion. Luke Kirk True 於 2023年5月13日 週六 上午1:30 寫道： > Hi David, > > For my own edification, when you refer to this change possibly putting > "more pressure on the brokers," is that from the downconversion of the >

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1845

See

[jira] [Created] (KAFKA-14996) CreateTopic falis with UnknownServerException if num partitions >= QuorumController.MAX_RECORDS_PER_BATCH

Edoardo Comar created KAFKA-14996: - Summary: CreateTopic falis with UnknownServerException if num partitions >= QuorumController.MAX_RECORDS_PER_BATCH Key: KAFKA-14996 URL:

Re: Unable to access JIRA

Hi Luke, I did self submission for the kafka contribution. Please check and guide me for the same. Thanks & Regards Ram On Fri, May 12, 2023 at 7:01 AM Luke Chen wrote: > Hi Ram, > > Please submit an account creation request and we'll approve it. >

Re: [DISCUSS] KIP-931: Flag to ignore unused message attribute field

Hi David, For my own edification, when you refer to this change possibly putting "more pressure on the brokers," is that from the downconversion of the message format, specifically, or something else? Thanks, Kirk On Fri, May 12, 2023, at 1:59 AM, Luke Chen wrote: > Hi David, > > I know what

Re: [DISCUSS] Adding non-committers as Github collaborators

Hey folks, Thanks for driving this initiative! I think the ability to assign reviewers / apply labels to PRs and re-trigger Jenkins builds is really useful and will also allow us to help out the community a bit more. Thanks, Yash On Fri, May 12, 2023 at 9:24 PM John Roesler wrote: > Thanks

[GitHub] [kafka-site] jlprat commented on a diff in pull request #510: MINOR: Add collaborator policy

jlprat commented on code in PR #510: URL: https://github.com/apache/kafka-site/pull/510#discussion_r1192560960 ## contributing.html: ## @@ -78,6 +78,20 @@ Becoming a Committer Demonstrated good understanding and exercised good technical judgement on at

Re: [DISCUSS] Adding non-committers as Github collaborators

Thanks again for bringing this up, David! As an update to the community, the PMC has approved a process to make use of this feature. Here are the relevant updates: PR to add the policy: https://github.com/apache/kafka-site/pull/510 PR to update the list:

[jira] [Created] (KAFKA-14995) Automate asf.yaml collaborators refresh

John Roesler created KAFKA-14995: Summary: Automate asf.yaml collaborators refresh Key: KAFKA-14995 URL: https://issues.apache.org/jira/browse/KAFKA-14995 Project: Kafka Issue Type:

[GitHub] [kafka-site] vvcephei commented on pull request #510: MINOR: Add collaborator policy

vvcephei commented on PR #510: URL: https://github.com/apache/kafka-site/pull/510#issuecomment-1545918143 cf https://github.com/apache/kafka/pull/13713 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [kafka-site] vvcephei commented on pull request #510: MINOR: Add collaborator policy

vvcephei commented on PR #510: URL: https://github.com/apache/kafka-site/pull/510#issuecomment-1545905343 For reference, this topic was initially proposed by @mumrah : https://lists.apache.org/thread/93lb6jhkjkmb9op9629xt6c6olwym28c -- This is an automated message from the Apache Git

[jira] [Created] (KAFKA-14994) jose4j is vulnerable to CVE- Improper Cryptographic Algorithm

Gaurav Jetly created KAFKA-14994: Summary: jose4j is vulnerable to CVE- Improper Cryptographic Algorithm Key: KAFKA-14994 URL: https://issues.apache.org/jira/browse/KAFKA-14994 Project: Kafka

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1844

See

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.4 #128

See Changes: -- [...truncated 526293 lines...] [2023-05-12T12:25:16.963Z] [2023-05-12T12:25:16.963Z] Gradle Test Run :core:integrationTest > Gradle Test Executor 167 >

[jira] [Created] (KAFKA-14993) Improve TransactionIndex instance handling while copying to and fetching from RSM.

Satish Duggana created KAFKA-14993: -- Summary: Improve TransactionIndex instance handling while copying to and fetching from RSM. Key: KAFKA-14993 URL: https://issues.apache.org/jira/browse/KAFKA-14993

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1843

See Changes: -- [...truncated 474203 lines...] [2023-05-12T09:56:35.584Z] [2023-05-12T09:56:35.584Z] Gradle Test Run :streams:integrationTest > Gradle Test Executor 179

Re: [DISCUSS] KIP-910: Update Source offsets for Source Connectors without producing records

Hi All, Thanks for the comments/reviews. I have updated the KIP https://cwiki.apache.org/confluence/display/KAFKA/KIP-910%3A+Update+Source+offsets+for+Source+Connectors+without+producing+records with a newer approach which shelves the need for an explicit topic. Please review again and let me

Re: [DISCUSS] KIP-926: introducing acks=min.insync.replicas config

That's fair so I leave it up to you, Luke. Cheers, David On Fri, May 12, 2023 at 10:58 AM Luke Chen wrote: > Hi David, > > Thanks for the response. > But I don't think the LEO-based leader election only benefit for this case. > Like in unclean clear election case, we now randomly chose a

Re: [DISCUSS] KIP-931: Flag to ignore unused message attribute field

Hi David, I know what you mean. Let's hear what others' thoughts about it. :) Luke On Fri, May 12, 2023 at 4:53 PM David Jacot wrote: > Thanks, Luke. > > > But if the producers and consumers all existed in the same organization, > which means upgrading producers/consumers for the org's cost

Re: [DISCUSS] KIP-926: introducing acks=min.insync.replicas config

Hi David, Thanks for the response. But I don't think the LEO-based leader election only benefit for this case. Like in unclean clear election case, we now randomly chose a out-of-sync replica to become the leader. This LEO-based leader election will help this case, too. Besides, not all producers

Re: [DISCUSS] KIP-931: Flag to ignore unused message attribute field

Thanks, Luke. > But if the producers and consumers all existed in the same organization, which means upgrading producers/consumers for the org's cost saving, should be a reasonable motivation. Yeah, that works in this case. However, Kafka is often used as a service (on premise or in cloud)

Re: [DISCUSS] KIP-926: introducing acks=min.insync.replicas config

Hi Luke, I disagree with this because we don't need the leader election change on its own if we don't do this KIP. They have to go together or not at all in my opinion. We need a KIP which designs the entire solution. Best, David On Fri, May 12, 2023 at 10:33 AM Luke Chen wrote: > Hi

Re: [DISCUSS] KIP-931: Flag to ignore unused message attribute field

Hi David, Yes, you're right. I've bumped the version of record batch, and describe the down-conversion will happen like what we do for message format v1 now when old consumers consuming records. > Overall, I wonder if the bandwidth saving is worth this change given that it will put more pressure

Re: [DISCUSS] KIP-926: introducing acks=min.insync.replicas config

Hi Alexandre, Thanks for the thoughts. I've thought about it, and think I would choose to have a new leader election method to fix the problem we encountered, not this "backup-only" replica solution. But this is still an interesting idea. Like what you've said, this solution can bring many

Re: [DISCUSS] KIP-926: introducing acks=min.insync.replicas config

Hi Haruki, Yes, this scenario could happen. I'm thinking we can fix it in step 6, when controller tried to get LEO from B,C replicas, the B,C replica should stop fetcher for this partition immediately, before returning the LEO. About if we need quorum-based or not, We can discuss in another KIP.

Re: [DISCUSS] KIP-926: introducing acks=min.insync.replicas config

Hi David, > It can't be in another KIP as it is required for your proposal to work. This is also an important part to discuss as it requires the controller to do more operations on leader changes. Yes, I know this is a requirement for this KIP to work, and need a lot of discussion. So that's why

Re: [DISCUSS] KIP-931: Flag to ignore unused message attribute field

Hi Luke, Thanks for the KIP. What do we do in the case where a batch is written with `ignoreMessageAttributes` set to 1, which means that messages won't have the `attributes`, and is consumed by a consumer which does not understand this new format? I suppose that we would need to introduce a new

[DISCUSS] KIP-931: Flag to ignore unused message attribute field

Hi all, I'd like to start a discussion for the KIP-931: Flag to ignore unused message attribute field. This KIP is to add a flag in the batch header to indicate if messages inside the batch have attribute field or not, to reduce the message size, thus, save network traffic and storage size (and