Hi all,
It turns out that there were other factors influencing my performance tests.
(actually hbase)Hence, more consumers than partitions in Flink was not the
problem. Thanks for the help!
On Wednesday, August 3, 2016 5:42 PM, neo21 zerro <neo21_ze...@yahoo.com>
wrote:
Hello,
I've tried to increase the network buffers but I didn't get any performance
improvement. However, I have to re-run some tests just to be sure that the
testing was not influenced by other factors. Will get back with more info.
Thanks for the help for now.
On Wednesday, August 3, 2016 12:58 PM, neo21 zerro <neo21_ze...@yahoo.com>
wrote:
It's the default, ProcessingTime.
On Wednesday, August 3, 2016 12:07 PM, Stephan Ewen <se...@apache.org>
wrote:
Hi!
Are you running on ProcessingTime or on EventTime?
Thanks,Stephan
On Wed, Aug 3, 2016 at 11:57 AM, neo21 zerro <neo21_ze...@yahoo.com> wrote:
Hi guys,
Thanks for getting back to me.
So to clarify:
Topology wise flink kafka source (does avro deserialization and small map)
-> window operator which does batching for 3 seconds -> hbase sink
Experiments:
1. flink source: parallelism 40 (20 idle tasks) -> window operator: parallelism
160 -> hbase sink: parallelism 160
- roughly 10.000 requests/sec on hbase
2. flink source: parallelism 20 -> window operator: parallelism 160 -> hbase
sink: parallelism 160
- roughly 100.000 requests/sec on hbase (10x more)
@Stephan as described below the parallelism of the sink was kept the same. I
agree with you that there is nothing to backpressue on the source ;) However,
my understanding right now is that only backpressure can be the explanation for
this situation. Since we just change the source parallelism, other things like
hbase parallelism are kept the same.
@Sameer all of those things are valid points. We make sure that we reduce the
row locking by partitioning the data on the hbase sinks. We are just after why
this limitation is happening. And since the same setup is used but just the
source parallelism is changed I don't expect this to be a hbase issue.
Thanks guys!
On Wednesday, August 3, 2016 11:38 AM, Sameer Wadkar <sam...@axiomine.com>
wrote:
What is the parallelism of the sink or the operator which writes to the sinks
in the first case. HBase puts are constrained by the following:
1. How your regions are distributed. Are you pre-splitting your regions for the
table. Do you know the number of regions your Hbase tables are split into.
2. Are all the sinks writing to all the regions. Meaning are you getting
records in the sink operator which could potentially go to any region. This can
become a big bottleneck if you have 40 sinks talking to all regions. I
pre-split my regions based on key salting and use custom partitioning to ensure
each sink operator writes to only a few regions and my performance shot up by
several orders.
3. You are probably doing this but adding puts in batches helps in general but
having each batch contain puts for too many regions hurts.
If the source parallelism is the same as the parallelism of other operators the
40 sinks communicating to all regions might be a problem. When you go down to
20 sinks you actually might be getting better performance due to lesser
resource contention on HBase.
Sent from my iPhone
> On Aug 3, 2016, at 4:14 AM, neo21 zerro <neo21_ze...@yahoo.com> wrote:
>
> Hello everybody,
>
> I'm using Flink Kafka consumer 0.8.x with kafka 0.8.2 and flink 1.0.3 on YARN.
> In kafka I have a topic which have 20 partitions and my flink topology reads
> from kafka (source) and writes to hbase (sink).
>
> when:
> 1. flink source has parallelism set to 40 (20 of the tasks are idle) I
>see 10.000 requests/sec on hbase
> 2. flink source has parallelism set to 20 (exact number of partitions) I
>see 100.0000 requests/sec on hbase (so a 10x improvement)
>
>
> It's clear that hbase is not the limiting factor in my topology.
> Assumption: Flink backpressure mechanism kicks in in the 1. case more
> aggressively and it's limiting the ingestion of tuples in the topology.
>
> The question: In the first case, why are those 20 sources which are sitting
> idle contributing so much to the backpressure?
>
>
> Thanks guys!
