Suggestion over architecture

Hello all, in my company we plan to set up the following architecture for our client: An internal kafka cluster in our company, and deploy a webapp (our software solution) on premise for our clients. We think to create one producer by "webapp" client in order to push in a global topic (in

Re: committing offset metadata in kafka streams

Sure. Sorry I was not clear. Thank you! lör 10 mars 2018 kl. 00:54 skrev Matthias J. Sax : > If there is only one partition by task, processing order is guaranteed. > > For default partitions grouper, it depends on your program. If you read > from multiple topics and

Re: Re: kafka steams with TimeWindows ,incorrect result

thx for your reply! I see that it is designed to operate on an infinite, unbounded stream of data. now I want to process for unbounded stream but divided by time interval . so what can I do for doing this ? funk...@live.com From: Guozhang

Re: [VOTE] 1.1.0 RC1

Great! Have a good weekend. On Fri, Mar 9, 2018 at 4:41 PM, Ismael Juma wrote: > Thanks Jeff: > > https://github.com/apache/kafka/pull/4678 > > Ismael > > On Fri, Mar 9, 2018 at 1:56 AM, Damian Guy wrote: > > > Hi Jeff, > > > > Thanks, we will look into

Re: [VOTE] 1.1.0 RC1

Thanks Jeff: https://github.com/apache/kafka/pull/4678 Ismael On Fri, Mar 9, 2018 at 1:56 AM, Damian Guy wrote: > Hi Jeff, > > Thanks, we will look into this. > > Regards, > Damian > > On Thu, 8 Mar 2018 at 18:27 Jeff Chao wrote: > > > Hello, > > >

Re: committing offset metadata in kafka streams

If there is only one partition by task, processing order is guaranteed. For default partitions grouper, it depends on your program. If you read from multiple topics and join/merge them, a task gets multiple partitions (from different topics) assigned. -Matthias On 3/9/18 2:42 PM, Stas Chizhov

Re: committing offset metadata in kafka streams

> Also note, that the processing order might slightly differ if you process the same data twice Is this still a problem when default partition grouper is used (with 1 partition per task)? Thank you, Stanislav. 2018-03-09 3:19 GMT+01:00 Matthias J. Sax : > Thanks

Re: [DISCUSS] KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils

Sweet! I think this pretty much wraps up all the discussion points. I'll update the KIP with all the relevant aspects we discussed and call for a vote. I'll also comment on the TopologyTestDriver ticket noting this modular test strategy. Thanks, everyone. -John On Fri, Mar 9, 2018 at 10:57 AM,

Re: Delayed processing

Hi Wim, One off-the-cuff idea is that you maybe don't need to actually delay anonymizing the data. Instead, you can just create a separate pathway that immediately anonymizes the data. Something like this: (raw-input topic, GDPR retention period set) |\->[streams apps that needs non-anonymized

Re: kafka steams with TimeWindows ,incorrect result

Hi Jie, This is by design of Kafka Streams, please read this doc for more details (search for "outputs of the Wordcount application is actually a continuous stream of updates"): https://kafka.apache.org/0110/documentation/streams/quickstart Note this semantics applies for both windowed and

Re: Delayed processing

Hello Wim, Thanks for your explanations, it makes more sense to me know. I think your scenario may be better described as a "re-processing" case than a "delayed processing" case, since you are effectively processing the un-anonymised data once, sending results to the topic; and then later you

Unit Test with Compacted Topics having non consecutive offsets

Hi, I need to write unit tests with Compacted Topics in local cluster. Has anyone done something like that? Any tips/guidance will be much appreciated. Thanks Sirisha

Re: [DISCUSS] KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils

John, Sorry for the delayed response. Thanks for the KIP, I'm +1 on it, and I don't have any further comments on the KIP itself aside from the comments that others have raised. Regarding the existing MockProcessorContext and its removal in favor of the one added from this KIP, I'm actually in

kafka steams with TimeWindows ,incorrect result

Hi： I used TimeWindow for aggregate data in kafka. this is code snippet ; view.flatMap(new MultipleKeyValueMapper(client)).groupByKey(Serialized.with(Serdes.String(), Serdes.serdeFrom(new CountInfoSerializer(), new CountInfoDeserializer(

Re: WELCOME to users@kafka.apache.org

Hello, I would like to report you on the website access issue from our Organization LAN public IP: 182.74.52.154. Looks like, public IP: 182.74.52.154 has been blocked at your end causing access issue from our network users. Could you please help me to unblock it? Thanks in advance! --

Re: [VOTE] 1.1.0 RC1

Hi Jeff, Thanks, we will look into this. Regards, Damian On Thu, 8 Mar 2018 at 18:27 Jeff Chao wrote: > Hello, > > We at Heroku have run 1.1.0 RC1 through our normal performance and > regression test suite and have found performance to be comparable to 1.0.0. > > That