handle multiple requests

[נתי אלמגור]

hello
i'm very new in Kafka , i have experience in Rabbit MQ 
i have connection Layer which publish to Rabbit queue request and worker app 
which have number of threads(for example 8 equal to CPU's number) that 
subscribe to Rabbit Queue and each request handled with one thread 

i can not  find  this solution in kafka  please help me   

Re: handle multiple requests

[Ricardo Ferreira]

Hi there,

Unless you are dealing with a low volume scenario, you should avoid tie 
each message/record to a specific thread. It will limit your ability to 
scale the processing out as CPU is a scarce resource. Alternatively, you 
should write your code to fetch multiple records at once (like a batch) 
and process them using the same thread that performed the fetch -- or 
hand over to another thread to decouple consumption from processing. 
Regardless, each thread should handle way more than just one record at a 
time. You can still control how each record is deemed processed by 
handling each offset individually.


This link  
contains a good intro about Kafka's consumer API.


Thanks,

-- Ricardo

On 6/10/20 10:20 AM, נתי אלמגור wrote:

hello
i'm very new in Kafka , i have experience in Rabbit MQ
i have connection Layer which publish to Rabbit queue request and worker app 
which have number of threads(for example 8 equal to CPU's number) that 
subscribe to Rabbit Queue and each request handled with one thread

i can not  find  this solution in kafka  please help me

Re: handle multiple requests

[Robin Moffatt]

You might also find this resource useful in general:
https://www.confluent.io/whitepaper/comparing-confluent-platform-with-traditional-messaging-middleware/


-- 

Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff


‪On Wed, 10 Jun 2020 at 16:00, ‫נתי אלמגור‬‎ 
wrote:‬

> hello
> i'm very new in Kafka , i have experience in Rabbit MQ
> i have connection Layer which publish to Rabbit queue request and worker
> app which have number of threads(for example 8 equal to CPU's number) that
> subscribe to Rabbit Queue and each request handled with one thread
>
> i can not  find  this solution in kafka  please help me
>

Re: Last Stable Offset (LSO) stuck for specific topic partition after Broker issues

[D C]

Hey peeps,

Anyone else encountered this and got to the bottom of it?

I'm facing a similar issue, having LSO stuck for some partitions in a topic
and the consumers can't get data out of it (we're using read_committed =
true).

When this issue started happening we were on kafka 2.3.1
i tried:
- restarting the consumers
- deleting the partition from the leader and letting it get in sync with
the new leader
- rolling restart of the brokers
- shutting down the whole cluster and starting it again
- tried deleting the txnindex files (after backing them up) and restarting
the brokers
- tried putting down the follower brokers of a partition and resyncing that
partition on them from scratch
- upgraded both kafka broker and client to 2.5.0

Now the following questions arise:
Where is the LSO actually stored (even if you get rid of the txnfiles, the
LSO stays the same).
Is there any way that the LSO can be reset?
Is there any way that you can manually abort and clean the state of a stuck
transaction? (i suspect that this is the reason why the LSO is stuck)
Is there any way to manually trigger a consistency check on the logfiles
that would fix any existing issues with either the logs or the indexes in
the partition?

Cheers,
Dragos

Kafka offset is discontinuous

[13729560619]

Hello,
 Is there congiurations to make kafka offset continuous? Help us !!! Spark 
cannot  skip the gap offset.

| |
13729560619
|
|
邮箱：13729560...@163.com
|

签名由 网易邮箱大师 定制