hi,all:
when I run stream application for a few minutes ,I got this error :
17/06/22 10:34:56 INFO ConsumerCoordinator: Revoking previously assigned
partitions [comment-0, profile-1, profile-3, cwb-3, bizs-1, cwb-1,
weibocomment-0, bizs-2, pages-0, bizs-4, pages-2, weibo-0, pages-4, weibo-4,
Hello,
I have the same problem with Kafka 0.10.1.0, but MirrorMaker is not replicating
anything, without any error message.
I’ve been scratching my head for a demi-dozen of hours now, and I can’t think
of what’s going on with my setup, my hundreds of topics keep unmirrored to my
destination
Karan, have you got auto.offset.reset in your consumer.properties?
https://kafka.apache.org/documentation/#newconsumerconfigs
On Thu, Jun 22, 2017 at 2:00 AM, karan alang wrote:
> Hi All,
>
> I've 2 Kafka clusters (Kafka 10) & I'm trying to test the MirrorMaker
>
I have been thinking about reducing all these overloaded functions for
stateful operations (there are some other places that introduces overloaded
functions but let's focus on these only in this discussion), what I used to
have is to use some "materialize" function on the KTables, like:
Thanks for the updated KIP, some more comments:
1.The config name is "default.deserialization.exception.handler" while the
interface class name is "RecordExceptionHandler", which is more general
than the intended purpose. Could we rename the class name accordingly?
2. Could you describe the full
Getting good concurrency in a webapp is more than doable. Check out these
benchmarks:
https://www.techempower.com/benchmarks/#section=data-r14=ph=db
I linked to the single query one because thats closest to a single
operation like you will be doing.
I'd also note if the data delivery does not
Hi All,
I've 2 Kafka clusters (Kafka 10) & I'm trying to test the MirrorMaker
functionality.
Here is what i did :
1) I have identical topics Topic1 on 2 Kafka clusters - Cluster1 & Cluster2
2) On Cluster1, I publish 100 messages on Topic1
3) I've 2 consumers reading messages from the 2 topics
If you are compressing messages than size of "compressed" message should be
less than what's specified in these parameters.
On Sat, Jun 17, 2017 at 7:46 PM, Eli Jordan
wrote:
> Hi
>
> max.message.bytes controls the maximum message size the kafka server will
> process
>
I appreciate everyone's feedback so far on this KIP.
Before starting a vote, I'd like to also ask for feedback on the
"Additional Food for Thought" section in the KIP:
That looks better than mine, nice! I think the tooling matters a lot to the
usability of the product we're shipping, being able to test out Kafka's
features on your own hardware/setup is very important to knowing if it can
work.
On Wed, Jun 21, 2017 at 8:01 PM, Apurva Mehta
Hi Tom,
I actually made modifications to the produce performance tool to do real
transactions earlier this week as part of our benchmarking (results
published here: bit.ly/kafka-eos-perf). I just submitted that patch here:
https://github.com/apache/kafka/pull/3400/files
I think my version is
Hi All - here is the update on this.
I was able to fix the following warnings -
1) WARN Property bootstrap.servers is not valid
(kafka.utils.VerifiableProperties)
-> removed bootstrap.servers from mmConsumer.config (IT IS REQUIRED
ONLY IN mmProducer.config)
2) zk.connectiontimeout.ms is
Hello All -
I've *Kafka 0.9* & I'm running this command to publish records to Kafka
topics -
$KAFKA_HOME/bin/kafka-verifiable-producer.sh --topic mmtopic1
--max-messages 500 --broker-list
localhost:9092,localhost:9093,localhost:9094,localhost:9095
--producer.config
I’m not really familiar with Netty so I won’t be of much help. Maybe try
posting on a Netty forum to see what they think?
-Dave
From: SenthilKumar K [mailto:senthilec...@gmail.com]
Sent: Wednesday, June 21, 2017 10:28 AM
To: Tauzell, Dave
Cc: users@kafka.apache.org; senthilec...@apache.org;
Hi there,
I'm -1 (non-binding) on shipping this RC.
Heroku has carried on performance testing with 0.11 RC1. We have updated
our test setup to use 0.11.0.0 RC1 client libraries. Without any of the
transactional features enabled, we get slightly better performance than
0.10.2.1 with 10.2.1 client
To make it clear, it’s outlined by Damian, I just copy pasted what he told me
in person :)
Eno
> On Jun 21, 2017, at 4:40 PM, Bill Bejeck wrote:
>
> +1 for the approach outlined above by Eno.
>
> On Wed, Jun 21, 2017 at 11:28 AM, Damian Guy wrote:
>
+1 for the approach outlined above by Eno.
On Wed, Jun 21, 2017 at 11:28 AM, Damian Guy wrote:
> Thanks Eno.
>
> Yes i agree. We could apply this same approach to most of the operations
> where we have multiple overloads, i.e., we have a single method for each
> operation
So netty would work for this case ? I do have netty server and seems to be
i'm not getting the expected results .. here is the git
https://github.com/senthilec566/netty4-server , is this right
implementation ?
Cheers,
Senthil
On Wed, Jun 21, 2017 at 7:45 PM, Tauzell, Dave
Thanks Eno.
Yes i agree. We could apply this same approach to most of the operations
where we have multiple overloads, i.e., we have a single method for each
operation that takes the required parameters and everything else is
specified as you have done above.
On Wed, 21 Jun 2017 at 16:24 Eno
(cc’ing user-list too)
Given that we already have StateStoreSuppliers that are configurable using the
fluent-like API, probably it’s worth discussing the other examples with joins
and serdes first since those have many overloads and are in need of some TLC.
So following your example, I guess
I see.
1. You don’t want the 100k machines sending directly to kafka.
2. You can only have a small number of web servers
People certainly have web-servers handling over 100k concurrent connections.
See this for some examples: https://github.com/smallnest/C1000K-Servers .
It
Thanks Jeyhun. Yes http server would be problematic here w.r.t network ,
memory ..
Hi Dave , The problem is not with Kafka , it's all about how do you handle
huge data before kafka. I did a simple test with 5 node Kafka Cluster
which gives good result ( ~950 MB/s ) ..So Kafka side i dont see a
What are your configurations?
- production
- brokers
- consumers
Is the problem that web servers cannot send to Kafka fast enough or your
consumers cannot process messages off of kafka fast enough?
What is the average size of these messages?
-Dave
-Original Message-
From: SenthilKumar
Hi,
With kafka you can increase overall throughput by increasing the number of
nodes in a cluster.
I had a similar issue, where we needed to ingest vast amounts of data to
streaming system.
In our case, kafka was a bottleneck, because of disk I/O. To solve it, we
implemented (simple) distributed
Hi Team , Sorry if this question is irrelevant to Kafka Group ...
I have been trying to solve problem of handling 5 GB/sec ingestion. Kafka
is really good candidate for us to handle this ingestion rate ..
100K machines > { Http Server (Jetty/Netty) } --> Kafka Cluster..
I see the problem
Thanks Guozhang,
I’ve updated the KIP and hopefully addressed all the comments so far. In the
process also changed the name of the KIP to reflect its scope better:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-161%3A+streams+deserialization+exception+handlers
If your business flow involves human actions, personally I would look at
a business process engine like the open source camunda.
Even if you don't choose to use it in production, you can use it to
prototype and evolve your design at the inception stage.
There's a simple to run example that
27 matches
Mail list logo