Re: Problem with kafka-streams aggregate windowedBy

2019-01-27 Thread Vincenzo D'Amore
m() > > >>>>> .map({ k, v -> > > >>>>> new KeyValue<>(k.window().end(), v) > > >>>>> }) > > >>>>> .to('output') > > >>>>> > > >>>>> def config = new Properties() > > >>>>> config.put(StreamsConfig.APPLICATION_ID_CONFIG, applicationId) > > >>>>> config.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, > > 'localhost:9092') > > >>>>> config.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, > > >>> TimeUnit.SECONDS.toMillis(60)) > > >>>>> > > >>>>> KafkaStreams kafkaStreams = new KafkaStreams(builder.build(), > > config) > > >>>>> kafkaStreams.start() > > >>>>> > > >>>>> > > >>>> I've tried add to config ConsumerConfig.AUTO_OFFSET_RESET_CONFIG set > > to > > >>>> 'latest' and 'earliest' but it didn't help. > > >>>> Can you help me understand what I'm doing wrong? > > >>>> Thank you. > > >>> > > > > > > > > -- Vincenzo D'Amore

Re: Kafka streams messages duplicates with non-overlapping gap-less windows

2019-01-27 Thread Vincenzo D'Amore
> https://www.confluent.io/blog/watermarks-tables-event-time-dataflow-model/ > > > -Matthias > > On 1/25/19 9:31 AM, Vincenzo D'Amore wrote: > > Hi all, > > > > I write here because it's a couple of days I'm struggling trying to > > understand why I've so mu

Kafka streams messages duplicates with non-overlapping gap-less windows

2019-01-25 Thread Vincenzo D'Amore
problem: https://github.com/westec/ks-aggregate-debug Given that I am using non-overlapping gap-less windows in kstream, the correct output should NOT contain duplicate messages between windows? Any ideas why the duplicates? -- Vincenzo D'Amore

Re: parallel processing of records in a Kafka consumer

2017-11-24 Thread Vincenzo D'Amore
ed over all running consumer > automatically for this case. > > > -Matthias > > On 11/22/17 12:22 PM, cours.syst...@gmail.com wrote: > > I am testing a KafkaConsumer. How can I modify it to process records in > parallel? > > > > -- Vincenzo D'Amore

Apache Kafka Cluster Setup

2017-10-24 Thread Vincenzo D'Amore
Hi all, I want setup a Kafka cluster in a production environment. During latest years I've worked with Solr user and, comparing the Kafka with Solr, it would be wonderful if even Kafka had an administration console where see what's happening. Looking around I've found this:

Kafka consumer - end of topic detection

2017-07-14 Thread Vincenzo D'Amore
at end of topic. I'm curious to know your suggestion, if check for a special closure message at end of topic is enough. Or if there are already best practices useful to handle this type of scenario. Best regards and again thanks in advance for your time, Vincenzo -- Vincenzo D'Amore email: v.dam

Re: Suggestions

2016-12-03 Thread Vincenzo D'Amore
> How long is 'forever'? Did you wait more than 5 minutes? > > On Fri, Dec 2, 2016 at 2:55 AM, Vincenzo D'Amore <v.dam...@gmail.com> > wrote: > > > Hi Kafka Gurus :) > > > > I'm creating process between few applications. > > > > First application crea

Suggestions

2016-12-02 Thread Vincenzo D'Amore
: "org.apache.kafka.common.serialization.StringDeserializer", "value.deserializer": "org.apache.kafka.common.serialization.StringDeserializer" usually there are 2 active groups (group_A and group_B). Best regards, Vincenzo -- Vincenzo D'Amore email: v.dam...@gmail.com skype: free.dev mobile: +39 349 8513251