Re: [PROPOSAL] Add commercial support page on website
FWIW - even if it's rejected by the Kafka PMC, you can maintain your own page for such information and provide your personal comments on them. If the object is to provide information and help users to make decisions, it should help. Although you should do the SEO by yourself, if the information is somehow neutral and valuable, you can ask the @apachekafka Twitter (X) account to propagate it and provide a blog for Kafka blogs. This is the common way how third-party "evangelist" producing content and get it promoted. Best, tison. Matthias J. Sax 于2024年1月13日周六 07:35写道： > > François, > > thanks for starting this initiative. Personally, I don't think it's > necessarily harmful for the project to add such a new page, however, I > share the same concerns others raised already. > > I understand your motivation that people had issues finding commercial > support, but I am not sure we can address this issue that way. I am also > "worried" (for the lack of a better word) that the page might become > long an unwieldy. In the end, any freelancer/consultant offering Kafka > services would be able to get on the page, so we might get hundreds of > entries, what also makes it impossible for users to find what they are > looking for. Also, the services of different companies might vary > drastically; should users read all these descriptions? I can also > imagine that some companies offer their services only in some > countries/regions making it even harder for user to find what they are > looking for? > > Overall, it sounds more like a search optimization problem, and thus it > seems out-of-scope what we can solve. As I said, I am not strictly > against it, but I just don't see much value either. > > > -Matthias > > On 1/11/24 12:55 PM, Francois Papon wrote: > > Hi Justine, > > > > You're right, Kafka is a part of my business (training, consulting, > > architecture design, sla...) and most of the time, users/customers said > > that it was hard for them to find a commercial support (in France for my > > case) after searching on the Kafka website (Google didn't help them). > > > > As an ASF member and PMC of several ASF projects, I know that this kind > > of page exist so this is why I made this proposal for the Kafka project > > because I really think that it can help users. > > > > As you suggest, I can submit a PR to be added on the "powered by" page. > > > > Thanks, > > > > François > > > > On 11/01/2024 21:00, Justine Olshan wrote: > >> Hey François, > >> > >> My point was that the companies on that page use kafka as part of their > >> business. If you use Kafka as part of your business feel free to submit a > >> PR to be added. > >> > >> I second Chris's point that other projects are not enough to require > >> Kafka > >> having such a support page. > >> > >> Justine > >> > >> On Thu, Jan 11, 2024 at 11:57 AM Chris Egerton > >> wrote: > >> > >>> Hi François, > >>> > >>> Is it an official policy of the ASF that projects provide a listing of > >>> commercial support options for themselves? I understand that other > >>> projects > >>> have chosen to provide one, but this doesn't necessarily imply that all > >>> projects should do the same, and I can't say I find this point very > >>> convincing as a rebuttal to some of the good-faith concerns raised by > >>> the > >>> PMC and members of the community so far. However, if there's an official > >>> ASF stance on this topic, then I acknowledge that Apache Kafka should > >>> align > >>> with it. > >>> > >>> Best, > >>> > >>> Chris > >>> > >>> > >>> On Thu, Jan 11, 2024, 14:50 fpapon wrote: > >>> > >>>> Hi Justine, > >>>> > >>>> I'm not sure to see the difference between "happy users" and vendors > >>>> that advertise their products in some of the company list in the > >>>> "powered by" page. > >>>> > >>>> Btw, my initial purpose of my proposal was to help user to find support > >>>> for production stuff rather than searching in google. > >>>> > >>>> I don't think this is a bad thing because this is something that > >>>> already > >>>> exist in many ASF projects like: > >>>> > >>>> https://hop.apache.org/community/commercial/ > >>>> https
Re: [QUESTION] What is the difference between sequence and offset for a Record?
Thanks for your reply! I may not use "normalization". What I want to refer to is: appendInfo.setLastOffset(offset.value - 1) which underneath updates the base offset field (in record batch) but not the offset delta of each record. Best, tison. Justine Olshan 于2023年8月8日周二 00:43写道： > The sequence summary looks right to me. > For log normalization, are you referring to compaction? The segment's first > and last offsets might change, but a batch keeps its offsets when > compaction occurs. > > Hope that helps. > Justine > > On Mon, Aug 7, 2023 at 8:59 AM Matthias J. Sax wrote: > > > > but the base offset may change during log normalizing. > > > > Not sure what you mean by "normalization" but offsets are immutable, so > > they don't change. (To be fair, I am not an expert on brokers, so not > > sure how this work in detail when log compaction ticks in). > > > > > This field is given by the producer and the broker should only read it. > > > > Sounds right. The point being is, that the broker has an "expected" > > value for it, and if the provided value does not match the expected one, > > the write is rejected to begin with. > > > > > > -Matthias > > > > On 8/7/23 6:35 AM, tison wrote: > > > Hi Matthias and Justine, > > > > > > Thanks for your reply! > > > > > > I can summarize the answer as - > > > > > > Record offset = base offset + offset delta. This field is calculated by > > the > > > broker and the delta won't change but the base offset may change during > > log > > > normalizing. > > > Record sequence = base sequence + (offset) delta. This field is given > by > > > the producer and the broker should only read it. > > > > > > Is it correct? > > > > > > I implement the manipulation part of base offset following this > > > understanding at [1]. > > > > > > Best, > > > tison. > > > > > > [1] > > > > > > https://github.com/tisonkun/kafka-api/blob/d080ab7e4b57c0ab0182e0b254333f400e616cd2/simplesrv/src/lib.rs#L391-L394 > > > > > > > > > Justine Olshan 于2023年8月2日周三 04:19写道： > > > > > >> For what it's worth -- the sequence number is not calculated > > >> "baseOffset/baseSequence + offset delta" but rather by monotonically > > >> increasing for a given epoch. If the epoch is bumped, we reset back to > > >> zero. > > >> This may mean that the offset and sequence may match, but do not > > strictly > > >> need to be the same. The sequence number will also always come from > the > > >> client and be in the produce records sent to the Kafka broker. > > >> > > >> As for offsets, there is some code in the log layer that maintains the > > log > > >> end offset and assigns offsets to the records. The produce handling on > > the > > >> leader should typically assign the offset. > > >> I believe you can find that code here: > > >> > > >> > > > https://github.com/apache/kafka/blob/b9a45546a7918799b6fb3c0fe63b56f47d8fcba9/core/src/main/scala/kafka/log/UnifiedLog.scala#L766 > > >> > > >> Justine > > >> > > >> On Tue, Aug 1, 2023 at 11:38 AM Matthias J. Sax > > wrote: > > >> > > >>> The _offset_ is the position of the record in the partition. > > >>> > > >>> The _sequence number_ is a unique ID that allows broker to > de-duplicate > > >>> messages. It requires the producer to implement the idempotency > > protocol > > >>> (part of Kafka transactions); thus, sequence numbers are optional and > > as > > >>> long as you don't want to support idempotent writes, you don't need > to > > >>> worry about them. (If you want to dig into details, checkout KIP-98 > > that > > >>> is the original KIP about Kafka TX). > > >>> > > >>> HTH, > > >>> -Matthias > > >>> > > >>> On 8/1/23 2:19 AM, tison wrote: > > >>>> Hi, > > >>>> > > >>>> I'm wringing a Kafka API Rust codec library[1] to understand how > Kafka > > >>>> models its concepts and how the core business logic works. > > >>>> > > >>>> During implementing the codec for Records[2], I saw a twins of > fields > > >>>> "sequence" and "offset". Both of them are calculated by > > >>>> baseOffset/baseSequence + offset delta. Then I'm a bit confused how > to > > >>> deal > > >>>> with them properly - what's the difference between these two > concepts > > >>>> logically? > > >>>> > > >>>> Also, to understand how the core business logic works, I write a > > simple > > >>>> server based on my codec library, and observe that the server may > need > > >> to > > >>>> update offset for records produced. How does Kafka set the correct > > >> offset > > >>>> for each produced records? And how does Kafka maintain the > calculation > > >>> for > > >>>> offset and sequence during these modifications? > > >>>> > > >>>> I'll appreciate if anyone can answer the question or give some > > insights > > >>> :D > > >>>> > > >>>> Best, > > >>>> tison. > > >>>> > > >>>> [1] https://github.com/tisonkun/kafka-api > > >>>> [2] https://kafka.apache.org/documentation/#messageformat > > >>>> > > >>> > > >> > > > > > >
Re: [QUESTION] What is the difference between sequence and offset for a Record?
Hi Matthias and Justine, Thanks for your reply! I can summarize the answer as - Record offset = base offset + offset delta. This field is calculated by the broker and the delta won't change but the base offset may change during log normalizing. Record sequence = base sequence + (offset) delta. This field is given by the producer and the broker should only read it. Is it correct? I implement the manipulation part of base offset following this understanding at [1]. Best, tison. [1] https://github.com/tisonkun/kafka-api/blob/d080ab7e4b57c0ab0182e0b254333f400e616cd2/simplesrv/src/lib.rs#L391-L394 Justine Olshan 于2023年8月2日周三 04:19写道： > For what it's worth -- the sequence number is not calculated > "baseOffset/baseSequence + offset delta" but rather by monotonically > increasing for a given epoch. If the epoch is bumped, we reset back to > zero. > This may mean that the offset and sequence may match, but do not strictly > need to be the same. The sequence number will also always come from the > client and be in the produce records sent to the Kafka broker. > > As for offsets, there is some code in the log layer that maintains the log > end offset and assigns offsets to the records. The produce handling on the > leader should typically assign the offset. > I believe you can find that code here: > > https://github.com/apache/kafka/blob/b9a45546a7918799b6fb3c0fe63b56f47d8fcba9/core/src/main/scala/kafka/log/UnifiedLog.scala#L766 > > Justine > > On Tue, Aug 1, 2023 at 11:38 AM Matthias J. Sax wrote: > > > The _offset_ is the position of the record in the partition. > > > > The _sequence number_ is a unique ID that allows broker to de-duplicate > > messages. It requires the producer to implement the idempotency protocol > > (part of Kafka transactions); thus, sequence numbers are optional and as > > long as you don't want to support idempotent writes, you don't need to > > worry about them. (If you want to dig into details, checkout KIP-98 that > > is the original KIP about Kafka TX). > > > > HTH, > >-Matthias > > > > On 8/1/23 2:19 AM, tison wrote: > > > Hi, > > > > > > I'm wringing a Kafka API Rust codec library[1] to understand how Kafka > > > models its concepts and how the core business logic works. > > > > > > During implementing the codec for Records[2], I saw a twins of fields > > > "sequence" and "offset". Both of them are calculated by > > > baseOffset/baseSequence + offset delta. Then I'm a bit confused how to > > deal > > > with them properly - what's the difference between these two concepts > > > logically? > > > > > > Also, to understand how the core business logic works, I write a simple > > > server based on my codec library, and observe that the server may need > to > > > update offset for records produced. How does Kafka set the correct > offset > > > for each produced records? And how does Kafka maintain the calculation > > for > > > offset and sequence during these modifications? > > > > > > I'll appreciate if anyone can answer the question or give some insights > > :D > > > > > > Best, > > > tison. > > > > > > [1] https://github.com/tisonkun/kafka-api > > > [2] https://kafka.apache.org/documentation/#messageformat > > > > > >
[QUESTION] What is the difference between sequence and offset for a Record?
Hi, I'm wringing a Kafka API Rust codec library[1] to understand how Kafka models its concepts and how the core business logic works. During implementing the codec for Records[2], I saw a twins of fields "sequence" and "offset". Both of them are calculated by baseOffset/baseSequence + offset delta. Then I'm a bit confused how to deal with them properly - what's the difference between these two concepts logically? Also, to understand how the core business logic works, I write a simple server based on my codec library, and observe that the server may need to update offset for records produced. How does Kafka set the correct offset for each produced records? And how does Kafka maintain the calculation for offset and sequence during these modifications? I'll appreciate if anyone can answer the question or give some insights :D Best, tison. [1] https://github.com/tisonkun/kafka-api [2] https://kafka.apache.org/documentation/#messageformat
