Re: New Kafka Consumer : unknown member id
Hi, Which version of kafka client are you using? I can't find this error message in the source code. When googling this error message, it showed the error is in Kafka v0.9. Could you try to use the V3.0.0 and see if that issue still exist? Thank you. Luke On Thu, Oct 28, 2021 at 11:15 PM Kafka Life wrote: > Dear Kafka Experts > > We have set up a group.id (consumer ) = YYY > But when tried to connect to kafka instance : i get this error message. I > am sure this consumer (group id does not exist in kafka) .We user plain > text protocol to connect to kafka 2.8.0. Please suggest how to resolve this > issue. > > DEBUG AbstractCoordinator:557 - [Consumer clientId=X, groupId=YYY] > Attempt to join group failed due to unknown member id. >
Contributors list
Hi, I have created a ticket and would like to work on it. Documentation says "you need to be in the contributors list of Apache Kafka in order to be assigned to a JIRA ticket". Can I be added to the contributors list? Full name is Shylaja Kokoori Thanks, -Shylaja -
Re: MirorMaker 2 - Accessing State Store ChangeLogs
Hi All Anybody has inputs on my question below ? Regards, Neeraj > On 26 Oct 2021, at 7:12 am, Neeraj Vaidya > wrote: > > Hello Experts > Any comments on this ? > > Regards, > Neeraj > > >> On 24 Oct 2021, at 11:55 am, Neeraj Vaidya wrote: >> >> Hi All, >> In an Active-Active MM2 (MirrorMaker 2) setup, I have the following Data >> Centres (DC1,DC2) : >> >> DC1: >> Topic T1 >> Kafka Streams which consumes from T1 and updates a local state store >> (example : "MyStateStore"). the applicationId is "myapp". >> >> DC2: >> Topic T1 >> The MM2 process replicates the records from T1 to DC1.T1 on this data centre. >> MM2 also replicates myapp-MyStateStore-changelog to >> DC1.myapp-MyStateStore-changelog. >> >> MM2 configuration: >> "sync.group.offsets.enabled = true". >> >> Questions : >> When I failover "myapp" to the DC2, how will the application get access to >> the State Store ? In my code I can only specify the name of the statestore >> from which to consume. >> Is there a way I can get access to the State Store in this site as well ? >> My theory is that in DC2 will only have records in the change log topic >> called DC1.myapp-MyStateStore-changelog so all state is essentially lost >> when I failover to DC2. >> >> Regards. >> Neeraj >
Re: Kafka streams - message not materialized in intermediate topics
Hi Tomer, To dump the topology you can do `System.out.println(topology.describe().toString())`. But if you can post just the code that would be fine as well. I understand about the logs, one thing to do is grep out any sensitive information, but I get it if you can't do that. Thanks, Bill On Thu, Oct 28, 2021 at 1:23 PM Tomer Cohen wrote: > Hi Bill, > > Is there an easy way to dump the topology to share? > > The logs contain sensitive information, is there something else that can be > provided? > > Thanks, > > Tomer > > On Thu, Oct 28, 2021 at 12:23 PM Bill Bejeck > wrote: > > > Hi Tomer, > > > > Can you share your topology and any log files? > > > > Thanks, > > Bill > > > > On Thu, Oct 28, 2021 at 12:07 PM Tomer Cohen wrote: > > > > > Hi Bill/Matthias, > > > > > > Thanks for the replies. > > > > > > The issue is I never see a result, I have a log that shows the message > > > coming in, but the adder/subtractor is never invoked for it even though > > it > > > should. So no result gets published to the intermediate topic I have. > > > > > > Thanks, > > > > > > Tomer > > > > > > On Thu, Oct 28, 2021 at 11:57 AM Bill Bejeck > > > > wrote: > > > > > > > Tomer, > > > > > > > > As Matthias pointed out for a single, final result you need to use > the > > > > `suppress()` operator. > > > > > > > > But back to your original question, > > > > > > > > they are processed by the adder/subtractor and are not > > > > > materialized in the intermediate topics which causes them not to be > > > > > outputted in the final topic > > > > > > > > > > > > > Is the issue you never see a result or were you curious about the > > > > intermediate calculations? > > > > HTH, > > > > Bill > > > > > > > > On Thu, Oct 28, 2021 at 1:05 AM Matthias J. Sax > > > wrote: > > > > > > > > > For this case, you can call `aggregate(...).suppress()`. > > > > > > > > > > -Matthias > > > > > > > > > > On 10/27/21 12:42 PM, Tomer Cohen wrote: > > > > > > Hi Bill, > > > > > > > > > > > > Thanks for the prompt reply. > > > > > > > > > > > > Setting to 0 forces a no collection window, so if I get 10 > messages > > > to > > > > > > aggregate for example, it will send 10 updates. But I only want > to > > > > > publish > > > > > > the final state only. > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Tomer > > > > > > > > > > > > On Wed, Oct 27, 2021 at 2:10 PM Bill Bejeck > > > > > > > > > > > > > wrote: > > > > > > > > > > > >> Hi Tomer, > > > > > >> > > > > > >> From the description you've provided, it sounds to me like you > > > have a > > > > > >> stateful operation. > > > > > >> > > > > > >> The thing to keep in mind with stateful operations in Kafka > > Streams > > > is > > > > > that > > > > > >> every result is not written to the changelog and forwarded > > > downstream. > > > > > >> Kafka Streams uses a cache for stateful operations and it's only > > on > > > > > cache > > > > > >> flush either when it's full or when Kafka Streams commits (every > > 30 > > > > > seconds > > > > > >> by default) that Kafka Streams writes the results of the > stateful > > > > > >> operations to the changelog and forwards the records downstream > to > > > > other > > > > > >> processors. > > > > > >> > > > > > >> If you want every Kafka Streams to forward every record you'll > > need > > > to > > > > > set > > > > > >> the `StreamConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG` to 0. > > > > > >> > > > > > >> If I haven't understood your experience accurately can you > > provide a > > > > few > > > > > >> more details? > > > > > >> > > > > > >> Thanks, > > > > > >> Bill > > > > > >> > > > > > >> > > > > > >> On Wed, Oct 27, 2021 at 9:48 AM Tomer Cohen > > > > wrote: > > > > > >> > > > > > >>> Hello Kafka team, > > > > > >>> > > > > > >>> I am seeing an odd behavior when using kafka streams. > > > > > >>> > > > > > >>> During periods of heavier volumes, there are messages coming > in. > > > > > However, > > > > > >>> they do look like they are processed by the adder/subtractor > and > > > are > > > > > not > > > > > >>> materialized in the intermediate topics which causes them not > to > > be > > > > > >>> outputted in the final topic. > > > > > >>> > > > > > >>> Is there any way to debug this or log out when a message is > > dropped > > > > in > > > > > >> the > > > > > >>> stream and not processed for whatever reason? > > > > > >>> > > > > > >>> Thanks, > > > > > >>> > > > > > >>> Tomer > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
Re: Kafka streams - message not materialized in intermediate topics
Hi Bill, Is there an easy way to dump the topology to share? The logs contain sensitive information, is there something else that can be provided? Thanks, Tomer On Thu, Oct 28, 2021 at 12:23 PM Bill Bejeck wrote: > Hi Tomer, > > Can you share your topology and any log files? > > Thanks, > Bill > > On Thu, Oct 28, 2021 at 12:07 PM Tomer Cohen wrote: > > > Hi Bill/Matthias, > > > > Thanks for the replies. > > > > The issue is I never see a result, I have a log that shows the message > > coming in, but the adder/subtractor is never invoked for it even though > it > > should. So no result gets published to the intermediate topic I have. > > > > Thanks, > > > > Tomer > > > > On Thu, Oct 28, 2021 at 11:57 AM Bill Bejeck > > wrote: > > > > > Tomer, > > > > > > As Matthias pointed out for a single, final result you need to use the > > > `suppress()` operator. > > > > > > But back to your original question, > > > > > > they are processed by the adder/subtractor and are not > > > > materialized in the intermediate topics which causes them not to be > > > > outputted in the final topic > > > > > > > > > > Is the issue you never see a result or were you curious about the > > > intermediate calculations? > > > HTH, > > > Bill > > > > > > On Thu, Oct 28, 2021 at 1:05 AM Matthias J. Sax > > wrote: > > > > > > > For this case, you can call `aggregate(...).suppress()`. > > > > > > > > -Matthias > > > > > > > > On 10/27/21 12:42 PM, Tomer Cohen wrote: > > > > > Hi Bill, > > > > > > > > > > Thanks for the prompt reply. > > > > > > > > > > Setting to 0 forces a no collection window, so if I get 10 messages > > to > > > > > aggregate for example, it will send 10 updates. But I only want to > > > > publish > > > > > the final state only. > > > > > > > > > > Thanks, > > > > > > > > > > Tomer > > > > > > > > > > On Wed, Oct 27, 2021 at 2:10 PM Bill Bejeck > > > > > > > > > > wrote: > > > > > > > > > >> Hi Tomer, > > > > >> > > > > >> From the description you've provided, it sounds to me like you > > have a > > > > >> stateful operation. > > > > >> > > > > >> The thing to keep in mind with stateful operations in Kafka > Streams > > is > > > > that > > > > >> every result is not written to the changelog and forwarded > > downstream. > > > > >> Kafka Streams uses a cache for stateful operations and it's only > on > > > > cache > > > > >> flush either when it's full or when Kafka Streams commits (every > 30 > > > > seconds > > > > >> by default) that Kafka Streams writes the results of the stateful > > > > >> operations to the changelog and forwards the records downstream to > > > other > > > > >> processors. > > > > >> > > > > >> If you want every Kafka Streams to forward every record you'll > need > > to > > > > set > > > > >> the `StreamConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG` to 0. > > > > >> > > > > >> If I haven't understood your experience accurately can you > provide a > > > few > > > > >> more details? > > > > >> > > > > >> Thanks, > > > > >> Bill > > > > >> > > > > >> > > > > >> On Wed, Oct 27, 2021 at 9:48 AM Tomer Cohen > > > wrote: > > > > >> > > > > >>> Hello Kafka team, > > > > >>> > > > > >>> I am seeing an odd behavior when using kafka streams. > > > > >>> > > > > >>> During periods of heavier volumes, there are messages coming in. > > > > However, > > > > >>> they do look like they are processed by the adder/subtractor and > > are > > > > not > > > > >>> materialized in the intermediate topics which causes them not to > be > > > > >>> outputted in the final topic. > > > > >>> > > > > >>> Is there any way to debug this or log out when a message is > dropped > > > in > > > > >> the > > > > >>> stream and not processed for whatever reason? > > > > >>> > > > > >>> Thanks, > > > > >>> > > > > >>> Tomer > > > > >>> > > > > >> > > > > > > > > > > > > > > >
Re: Kafka streams - message not materialized in intermediate topics
Hi Tomer, Can you share your topology and any log files? Thanks, Bill On Thu, Oct 28, 2021 at 12:07 PM Tomer Cohen wrote: > Hi Bill/Matthias, > > Thanks for the replies. > > The issue is I never see a result, I have a log that shows the message > coming in, but the adder/subtractor is never invoked for it even though it > should. So no result gets published to the intermediate topic I have. > > Thanks, > > Tomer > > On Thu, Oct 28, 2021 at 11:57 AM Bill Bejeck > wrote: > > > Tomer, > > > > As Matthias pointed out for a single, final result you need to use the > > `suppress()` operator. > > > > But back to your original question, > > > > they are processed by the adder/subtractor and are not > > > materialized in the intermediate topics which causes them not to be > > > outputted in the final topic > > > > > > > Is the issue you never see a result or were you curious about the > > intermediate calculations? > > HTH, > > Bill > > > > On Thu, Oct 28, 2021 at 1:05 AM Matthias J. Sax > wrote: > > > > > For this case, you can call `aggregate(...).suppress()`. > > > > > > -Matthias > > > > > > On 10/27/21 12:42 PM, Tomer Cohen wrote: > > > > Hi Bill, > > > > > > > > Thanks for the prompt reply. > > > > > > > > Setting to 0 forces a no collection window, so if I get 10 messages > to > > > > aggregate for example, it will send 10 updates. But I only want to > > > publish > > > > the final state only. > > > > > > > > Thanks, > > > > > > > > Tomer > > > > > > > > On Wed, Oct 27, 2021 at 2:10 PM Bill Bejeck > > > > > > > wrote: > > > > > > > >> Hi Tomer, > > > >> > > > >> From the description you've provided, it sounds to me like you > have a > > > >> stateful operation. > > > >> > > > >> The thing to keep in mind with stateful operations in Kafka Streams > is > > > that > > > >> every result is not written to the changelog and forwarded > downstream. > > > >> Kafka Streams uses a cache for stateful operations and it's only on > > > cache > > > >> flush either when it's full or when Kafka Streams commits (every 30 > > > seconds > > > >> by default) that Kafka Streams writes the results of the stateful > > > >> operations to the changelog and forwards the records downstream to > > other > > > >> processors. > > > >> > > > >> If you want every Kafka Streams to forward every record you'll need > to > > > set > > > >> the `StreamConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG` to 0. > > > >> > > > >> If I haven't understood your experience accurately can you provide a > > few > > > >> more details? > > > >> > > > >> Thanks, > > > >> Bill > > > >> > > > >> > > > >> On Wed, Oct 27, 2021 at 9:48 AM Tomer Cohen > > wrote: > > > >> > > > >>> Hello Kafka team, > > > >>> > > > >>> I am seeing an odd behavior when using kafka streams. > > > >>> > > > >>> During periods of heavier volumes, there are messages coming in. > > > However, > > > >>> they do look like they are processed by the adder/subtractor and > are > > > not > > > >>> materialized in the intermediate topics which causes them not to be > > > >>> outputted in the final topic. > > > >>> > > > >>> Is there any way to debug this or log out when a message is dropped > > in > > > >> the > > > >>> stream and not processed for whatever reason? > > > >>> > > > >>> Thanks, > > > >>> > > > >>> Tomer > > > >>> > > > >> > > > > > > > > > >
Re: Kafka streams - message not materialized in intermediate topics
Hi Bill/Matthias, Thanks for the replies. The issue is I never see a result, I have a log that shows the message coming in, but the adder/subtractor is never invoked for it even though it should. So no result gets published to the intermediate topic I have. Thanks, Tomer On Thu, Oct 28, 2021 at 11:57 AM Bill Bejeck wrote: > Tomer, > > As Matthias pointed out for a single, final result you need to use the > `suppress()` operator. > > But back to your original question, > > they are processed by the adder/subtractor and are not > > materialized in the intermediate topics which causes them not to be > > outputted in the final topic > > > > Is the issue you never see a result or were you curious about the > intermediate calculations? > HTH, > Bill > > On Thu, Oct 28, 2021 at 1:05 AM Matthias J. Sax wrote: > > > For this case, you can call `aggregate(...).suppress()`. > > > > -Matthias > > > > On 10/27/21 12:42 PM, Tomer Cohen wrote: > > > Hi Bill, > > > > > > Thanks for the prompt reply. > > > > > > Setting to 0 forces a no collection window, so if I get 10 messages to > > > aggregate for example, it will send 10 updates. But I only want to > > publish > > > the final state only. > > > > > > Thanks, > > > > > > Tomer > > > > > > On Wed, Oct 27, 2021 at 2:10 PM Bill Bejeck > > > > wrote: > > > > > >> Hi Tomer, > > >> > > >> From the description you've provided, it sounds to me like you have a > > >> stateful operation. > > >> > > >> The thing to keep in mind with stateful operations in Kafka Streams is > > that > > >> every result is not written to the changelog and forwarded downstream. > > >> Kafka Streams uses a cache for stateful operations and it's only on > > cache > > >> flush either when it's full or when Kafka Streams commits (every 30 > > seconds > > >> by default) that Kafka Streams writes the results of the stateful > > >> operations to the changelog and forwards the records downstream to > other > > >> processors. > > >> > > >> If you want every Kafka Streams to forward every record you'll need to > > set > > >> the `StreamConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG` to 0. > > >> > > >> If I haven't understood your experience accurately can you provide a > few > > >> more details? > > >> > > >> Thanks, > > >> Bill > > >> > > >> > > >> On Wed, Oct 27, 2021 at 9:48 AM Tomer Cohen > wrote: > > >> > > >>> Hello Kafka team, > > >>> > > >>> I am seeing an odd behavior when using kafka streams. > > >>> > > >>> During periods of heavier volumes, there are messages coming in. > > However, > > >>> they do look like they are processed by the adder/subtractor and are > > not > > >>> materialized in the intermediate topics which causes them not to be > > >>> outputted in the final topic. > > >>> > > >>> Is there any way to debug this or log out when a message is dropped > in > > >> the > > >>> stream and not processed for whatever reason? > > >>> > > >>> Thanks, > > >>> > > >>> Tomer > > >>> > > >> > > > > > >
Re: Kafka streams - message not materialized in intermediate topics
Tomer, As Matthias pointed out for a single, final result you need to use the `suppress()` operator. But back to your original question, they are processed by the adder/subtractor and are not > materialized in the intermediate topics which causes them not to be > outputted in the final topic > Is the issue you never see a result or were you curious about the intermediate calculations? HTH, Bill On Thu, Oct 28, 2021 at 1:05 AM Matthias J. Sax wrote: > For this case, you can call `aggregate(...).suppress()`. > > -Matthias > > On 10/27/21 12:42 PM, Tomer Cohen wrote: > > Hi Bill, > > > > Thanks for the prompt reply. > > > > Setting to 0 forces a no collection window, so if I get 10 messages to > > aggregate for example, it will send 10 updates. But I only want to > publish > > the final state only. > > > > Thanks, > > > > Tomer > > > > On Wed, Oct 27, 2021 at 2:10 PM Bill Bejeck > > wrote: > > > >> Hi Tomer, > >> > >> From the description you've provided, it sounds to me like you have a > >> stateful operation. > >> > >> The thing to keep in mind with stateful operations in Kafka Streams is > that > >> every result is not written to the changelog and forwarded downstream. > >> Kafka Streams uses a cache for stateful operations and it's only on > cache > >> flush either when it's full or when Kafka Streams commits (every 30 > seconds > >> by default) that Kafka Streams writes the results of the stateful > >> operations to the changelog and forwards the records downstream to other > >> processors. > >> > >> If you want every Kafka Streams to forward every record you'll need to > set > >> the `StreamConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG` to 0. > >> > >> If I haven't understood your experience accurately can you provide a few > >> more details? > >> > >> Thanks, > >> Bill > >> > >> > >> On Wed, Oct 27, 2021 at 9:48 AM Tomer Cohen wrote: > >> > >>> Hello Kafka team, > >>> > >>> I am seeing an odd behavior when using kafka streams. > >>> > >>> During periods of heavier volumes, there are messages coming in. > However, > >>> they do look like they are processed by the adder/subtractor and are > not > >>> materialized in the intermediate topics which causes them not to be > >>> outputted in the final topic. > >>> > >>> Is there any way to debug this or log out when a message is dropped in > >> the > >>> stream and not processed for whatever reason? > >>> > >>> Thanks, > >>> > >>> Tomer > >>> > >> > > >
New Kafka Consumer : unknown member id
Dear Kafka Experts We have set up a group.id (consumer ) = YYY But when tried to connect to kafka instance : i get this error message. I am sure this consumer (group id does not exist in kafka) .We user plain text protocol to connect to kafka 2.8.0. Please suggest how to resolve this issue. DEBUG AbstractCoordinator:557 - [Consumer clientId=X, groupId=YYY] Attempt to join group failed due to unknown member id.
Re: Contributors list
Done. On 10/27/21 11:55 PM, Michael Negodaev wrote: Hello! Please add me to the contributor list in JIRA with username "mnegodaev". Thank you.
Contributors list
Hello! Please add me to the contributor list in JIRA with username "mnegodaev". Thank you.
