Thanks for the solution please help me to try out the keyed partitioning
with 1000 partitions.And also give suggestions to try Kafka with node js.
On Tue, 8 Jan 2019, 9:53 pm Todd Palino OK, in that case you’ll want to do something like use the sensor ID as the
> key of the message. This will ass
Latency sounds high to me, maybe your JVMs are GC'ing a lot?
Ryanne
On Tue, Jan 8, 2019, 11:45 AM Gioacchino Vino Hi expert,
>
>
> I would ask you some guidelines, web-pages or comments regarding my
> use-case.
>
>
> *Requirements*:
>
> - 2000+ producers
>
> - input rate 600k messages/s
>
> - co
For this case no reordering will occur, but the original send order will
be preserved.
This is even true without transactions if you use idempotent producer.
-Matthias
On 1/6/19 6:00 PM, Mark Horton wrote:
> I was curious if re-ordering is possible within a transaction? Let's
> say a KafkaProduc
Hi expert,
I would ask you some guidelines, web-pages or comments regarding my
use-case.
*Requirements*:
- 2000+ producers
- input rate 600k messages/s
- consumers must write in 3 different databases (so i assume 3 consumer
groups) at 600k messages/s overall (200k messages/s/database)
Just key your messages by user ID and they'll be processed in sequence.
Every message with the same key will go to the same partition, regardless
of how many partitions there are.
Ryanne
On Tue, Jan 8, 2019, 8:57 AM marimuthu eee Hi Ryanne Dolan,
>
> My requirement is to process notification mes
On 08.01.2019 17:11, aruna ramachandran wrote:
> I need to process single sensor messages in serial (order of messages
> should not be changed)at the same time I have to process 1 sensors
> messages in parallel please help me to configure the topics and partitions.
>
If you want to process e
Yes, As Todd said you have to use some id as the key to partition.
The rebalancing will be an over head and if you increase the partitions
later you will lose the order.
you can go through
https://anirudhbhatnagar.com/2016/08/22/achieving-order-guarnetee-in-kafka-with-partitioning/
for more unders
OK, in that case you’ll want to do something like use the sensor ID as the
key of the message. This will assure that every message for that sensor ID
ends up in the same partition (which will assure strict ordering of
messages for that sensor ID).
Then you can create a number of partitions to get
I need to process single sensor messages in serial (order of messages
should not be changed)at the same time I have to process 1 sensors
messages in parallel please help me to configure the topics and partitions.
On Tue, Jan 8, 2019 at 9:19 PM Todd Palino wrote:
> I think you’ll need to expa
Hi,
My requirement is to process notification messages in parallel.So i have an
idea to pick kafka as a messaging system.But i need dynamic partitioning to
handle individual user notification messages.Is it possible to create
dynamic partition creation for a particular topic.
I think you’ll need to expand a little more here and explain what you mean
by processing them in parallel. Nearly by definition, parallelization and
strict ordering are mutually exclusive concepts.
-Todd
On Tue, Jan 8, 2019 at 10:40 AM aruna ramachandran
wrote:
> I need to process the 1 sen
I need to process the 1 sensor messages in parallel but each sensor
message should be in order.If I create 1 partition it doesn't give high
throughput .Order is guaranteed only inside the partition. How can
parallelize messages without changing the order pls help me to find the
solution.
I would recommend taking a look at this presentation:
https://www.slideshare.net/ConfluentInc/how-much-kafka/ConfluentInc/how-much-kafka
At the end of the presentation, there are also some useful links.
Cheers!
--
Jonathan
On Tue, Jan 8, 2019 at 2:01 PM marimuthu eee
wrote:
> Hi,
>
> How man
Hi Ryanne Dolan,
My requirement is to process notification messages in parallel.So i have an
idea to pick kafka as a messaging system.But i need dynamic partitioning*
to handle individual user notification messages in serial manner*.Is it
possible to create dynamic partition creation for a particu
Added the log file (In the previous mail I saw the lines are cut)
On Tue, Jan 8, 2019 at 2:39 PM Nitay Kufert wrote:
> Thanks. it seems promising. Sounds a lot like the problems we are having.
> Do you know when the fix will be released?
>
> BTW, It just happened to us again, this time when I m
Can you explain why you think you need dynamic partitioning? Maybe we can
help you design without it.
Ryanne
On Tue, Jan 8, 2019, 4:04 AM marimuthu eee Hi,
>
> My requirement is to process notification messages in parallel.So i have an
> idea to pick kafka as a messaging system.But i need dynami
Hi,
How many amount of partition can put in kafka topic and How many amount of
topic can created in kafka..
Hi,
How many amount of partition can put in kafka topic.
Hello Marimuthu,
Partitions are created at the moment the topic is created, however, you can
also add more new partitions to a topic once have been created. There is
not a possibility to create partitions dynamically (out-of-the-box) into a
topic unless you create your own business logic and/or sc
On 1/8/19 12:57 PM, Peter Levart wrote:
Hi John,
On 1/8/19 12:45 PM, Peter Levart wrote:
I looked at your custom transfomer, and it looks almost correct to
me. The
only flaw seems to be that it only looks
for closed windows for the key currently being processed, which
means that
if you ha
Thanks. it seems promising. Sounds a lot like the problems we are having.
Do you know when the fix will be released?
BTW, It just happened to us again, this time when I manually added 2 new
instances (we had 4 and I increased to 6).
This is the compacted topic showing the data loss:
CreateTime:1
Hi John,
On 1/8/19 12:45 PM, Peter Levart wrote:
I looked at your custom transfomer, and it looks almost correct to
me. The
only flaw seems to be that it only looks
for closed windows for the key currently being processed, which means
that
if you have key "A" buffered, but don't get another e
Hi John,
On 1/7/19 9:10 PM, John Roesler wrote:
Hi Peter,
Sorry, I just now have seen this thread.
You asked if this behavior is unexpected, and the answer is yes.
Suppress.untilWindowCloses is intended to emit only the final result,
regardless of restarts.
You also asked how the suppression
My requirement is to process notification messages in parallel.So I have an
idea to pick Kafka as a messaging system.But I need dynamic partitioning to
handle individual user notification messages.Is it possible to create
dynamic partition creation for a particular topic.
Hi,
My requirement is to process notification messages in parallel.So i have an
idea to pick kafka as a messaging system.But i need dynamic partitioning to
handle individual user notification messages.Is it possible to create
dynamic partition creation for a particular topic.
Hello Guys,
Can you please provide some insights into how much Average GC usage is in
your Kafka brokers.
I am seeing really high GC usage in some of our brokers. Sometimes it gets
as high as 30% and our producers start lagging.
--
Thanks,
Ankur Rana
Software Developer
FarEye
26 matches
Mail list logo
