Hi Tomer, To dump the topology you can do `System.out.println(topology.describe().toString())`. But if you can post just the code that would be fine as well.
I understand about the logs, one thing to do is grep out any sensitive information, but I get it if you can't do that. Thanks, Bill On Thu, Oct 28, 2021 at 1:23 PM Tomer Cohen <ilan...@gmail.com> wrote: > Hi Bill, > > Is there an easy way to dump the topology to share? > > The logs contain sensitive information, is there something else that can be > provided? > > Thanks, > > Tomer > > On Thu, Oct 28, 2021 at 12:23 PM Bill Bejeck <b...@confluent.io.invalid> > wrote: > > > Hi Tomer, > > > > Can you share your topology and any log files? > > > > Thanks, > > Bill > > > > On Thu, Oct 28, 2021 at 12:07 PM Tomer Cohen <ilan...@gmail.com> wrote: > > > > > Hi Bill/Matthias, > > > > > > Thanks for the replies. > > > > > > The issue is I never see a result, I have a log that shows the message > > > coming in, but the adder/subtractor is never invoked for it even though > > it > > > should. So no result gets published to the intermediate topic I have. > > > > > > Thanks, > > > > > > Tomer > > > > > > On Thu, Oct 28, 2021 at 11:57 AM Bill Bejeck <b...@confluent.io.invalid > > > > > wrote: > > > > > > > Tomer, > > > > > > > > As Matthias pointed out for a single, final result you need to use > the > > > > `suppress()` operator. > > > > > > > > But back to your original question, > > > > > > > > they are processed by the adder/subtractor and are not > > > > > materialized in the intermediate topics which causes them not to be > > > > > outputted in the final topic > > > > > > > > > > > > > Is the issue you never see a result or were you curious about the > > > > intermediate calculations? > > > > HTH, > > > > Bill > > > > > > > > On Thu, Oct 28, 2021 at 1:05 AM Matthias J. Sax <mj...@apache.org> > > > wrote: > > > > > > > > > For this case, you can call `aggregate(...).suppress()`. > > > > > > > > > > -Matthias > > > > > > > > > > On 10/27/21 12:42 PM, Tomer Cohen wrote: > > > > > > Hi Bill, > > > > > > > > > > > > Thanks for the prompt reply. > > > > > > > > > > > > Setting to 0 forces a no collection window, so if I get 10 > messages > > > to > > > > > > aggregate for example, it will send 10 updates. But I only want > to > > > > > publish > > > > > > the final state only. > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Tomer > > > > > > > > > > > > On Wed, Oct 27, 2021 at 2:10 PM Bill Bejeck > > > <b...@confluent.io.invalid > > > > > > > > > > > wrote: > > > > > > > > > > > >> Hi Tomer, > > > > > >> > > > > > >> From the description you've provided, it sounds to me like you > > > have a > > > > > >> stateful operation. > > > > > >> > > > > > >> The thing to keep in mind with stateful operations in Kafka > > Streams > > > is > > > > > that > > > > > >> every result is not written to the changelog and forwarded > > > downstream. > > > > > >> Kafka Streams uses a cache for stateful operations and it's only > > on > > > > > cache > > > > > >> flush either when it's full or when Kafka Streams commits (every > > 30 > > > > > seconds > > > > > >> by default) that Kafka Streams writes the results of the > stateful > > > > > >> operations to the changelog and forwards the records downstream > to > > > > other > > > > > >> processors. > > > > > >> > > > > > >> If you want every Kafka Streams to forward every record you'll > > need > > > to > > > > > set > > > > > >> the `StreamConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG` to 0. > > > > > >> > > > > > >> If I haven't understood your experience accurately can you > > provide a > > > > few > > > > > >> more details? > > > > > >> > > > > > >> Thanks, > > > > > >> Bill > > > > > >> > > > > > >> > > > > > >> On Wed, Oct 27, 2021 at 9:48 AM Tomer Cohen <ilan...@gmail.com> > > > > wrote: > > > > > >> > > > > > >>> Hello Kafka team, > > > > > >>> > > > > > >>> I am seeing an odd behavior when using kafka streams. > > > > > >>> > > > > > >>> During periods of heavier volumes, there are messages coming > in. > > > > > However, > > > > > >>> they do look like they are processed by the adder/subtractor > and > > > are > > > > > not > > > > > >>> materialized in the intermediate topics which causes them not > to > > be > > > > > >>> outputted in the final topic. > > > > > >>> > > > > > >>> Is there any way to debug this or log out when a message is > > dropped > > > > in > > > > > >> the > > > > > >>> stream and not processed for whatever reason? > > > > > >>> > > > > > >>> Thanks, > > > > > >>> > > > > > >>> Tomer > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
