Re: [External] Re: Is order of data is not maintained in Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios such as Updating a Bank Account scenarios etc

If you use partitioning based on your client/customer ID, events for a particular customer will go to a unique partition and hence will always be ordered. This still allows processing events from different customers in parallel. Will also support scaling out to some degree. However, if you need ev

Re: Unclean Leader Election Expected Behavior

You can enable unclean.leader.election temporarily for specific topic by using kafka-topics.sh command. This requires broker restart to take effect. http://kafka.apache.org/documentation/#topicconfigs On Thu, Jun 28, 2018 at 2:27 AM Jordan Pilat wrote: > Heya, > > I had a question about what b

Re: [DISCUSS] KIP-328: Ability to suppress updates for KTables

Hi Guozhang, Thanks for the review and the alternative idea. A quick note on the metrics. I actually do think that it should be true that "Skipped records are records that are for one reason or another invalid." I recently added the change to record a skipped-record when we get a record for a wi

RE: [External] Re: Is order of data is not maintained in Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios such as Updating a Bank Account scenarios etc

Yes, that is correct.. I guess updating a Bank Account should be done in sequence not parallel.. the reason use Kafka is maybe the better throughput compare to other message queues? -Original Message- From: Malik, Shibha (GE Renewable Energy, consultant) [mailto:shibha.ma...@ge.com]

RE: [External] Re: Is order of data is not maintained in Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios such as Updating a Bank Account scenarios etc

But then restricting a consumer to use only partition seems to be similar to traditional message queues ( except the persistence feature ) and does not offer parallelism , isn’t it ? -Original Message- From: Yi, Gene [mailto:gene...@sc.com.INVALID] Sent: Wednesday, June 27, 2018 5:43 PM

RE: [External] Re: Is order of data is not maintained in Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios such as Updating a Bank Account scenarios etc

I think you may use one partition only? __ Gene Huijian Yi TS - FM - App Standard Chartered Global Business Services Co., Ltd. Phone: +862259806930 Address:Standard Chartered Center, 3/F No.35 Xinhuanbei Road, TEDA

Re: Is order of data is not maintained in Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios such as Updating a Bank Account scenarios etc

Hi, Kafka guarantees ordering only within a partition (not across partitions in a topic) ie consumers reading off a partition will always get records in the order they were *received into the partition*. Note that in some cases of retries by a non- idempotent producer, the order in which a produc

Is order of data is not maintained in Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios such as Updating a Bank Account scenarios etc

Is order of data is not maintained in Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios such as Updating a Bank Account scenarios etc

What are the use cases where technologies like Kafka , Storm , Flink , , Hive , Hadoop and Spark differentiates ?

Hi all, What are the use cases where technologies like Kafka , Storm , Flink , , Hive , Hadoop and Spark differentiates ? Is there a good material online or book to refer for this ? Thanks, Shibha

Re: Can multiple producers write to the same partition ?

Yes. If you do this, the writes of both producers with interleave and there are no ordering guarantees between records written by different producers. -Matthias On 6/27/18 11:26 AM, Malik, Shibha (GE Renewable Energy, consultant) wrote: > Hi, > > Can multiple producers write to the same partit

Re: [DISCUSS] KIP-328: Ability to suppress updates for KTables

Hello John, thanks for putting up the KIP. I have a meta comment: We need to clarify the difference between late event suppression semantics and the window retention semantics that result into a windowed KTable. More specifically, say you have a window of size 10 min, and `until` 20 min, and the r

Unclean Leader Election Expected Behavior

Heya, I had a question about what behavior to expect from a particular scenario. Given: A. Unclean leader elections are disabled B. A partition is led by Broker1 and followed by Broker2 C. Broker1 is on offset 100 D. Broker2 is on offset 90 E. Broker2 has fallen out of the ISR, leaving only Brok

Re: [DISCUSS] KIP-328: Ability to suppress updates for KTables

Thanks for the feedback, Matthias, It seems like in straightforward relational processing cases, it would not make sense to bound the lateness of KTables. In general, it seems better to have "guard rails" in place that make it easier to write sensible programs than insensible ones. But I'm still

Re: [DISCUSS] KIP-328: Ability to suppress updates for KTables

Hello again all, I realized today that I neglected to include metrics in the proposal. I have added them just now. Thanks, -John On Tue, Jun 26, 2018 at 3:11 PM John Roesler wrote: > Hello devs and users, > > Please take some time to consider this proposal for Kafka Streams: > > KIP-328: Abili

Can multiple producers write to the same partition ?

Hi, Can multiple producers write to the same partition ?

Re: [DISCUSS] KIP-328: Ability to suppress updates for KTables

I noticed this (lack of primary parameter) as well. What you gave as new example is semantically the same as what I suggested. So it is good by me. Thanks On Wed, Jun 27, 2018 at 7:31 AM, John Roesler wrote: > Thanks for taking look, Ted, > > I agree this is a departure from the conventions of

Re: [DISCUSS] KIP-328: Ability to suppress updates for KTables

Thanks for taking look, Ted, I agree this is a departure from the conventions of Streams DSL. Most of our config objects have one or two "required" parameters, which fit naturally with the static factory method approach. TimeWindow, for example, requires a size parameter, so we can naturally say

Whats the Kafka Broker Behavior when ZooKeeperCluster is DOWN

Hi All, In a 3 Node Zookeeper cluster if 2 Nodes goes down but Zookeeper to which Kafka broker is up, what is expected ?? Is kafka borker expected to go down ?? or only the new operations will fail. Thanks Harish

Re: [kafka-clients] Re: [VOTE] 1.1.1 RC1

+1 non-binding Ran quick start and checked signatures On Wed, Jun 27, 2018 at 8:07 AM, Dong Lin wrote: > Thank you all for your test and votes! > > It will be great to have more votes from PMC so that we can conclude > kafka-1.1.1 release :)

Re: [kafka-clients] Re: [VOTE] 1.1.1 RC1

Thank you all for your test and votes! It will be great to have more votes from PMC so that we can conclude kafka-1.1.1 release :)