Hi Hawin!
The performance tuning of Kafka is much trickier than that of Flink. Your
performance bottleneck may be Kafka at this point, not Flink.
To make Kafka fast, make sure you have the right setup for the data
directories, and you set up zookeeper properly (for good throughput).
To test the
Hi Stephan
Yes, that is a great idea. if it is possible, I will try my best to
contribute some codes to Flink.
But I have to run some flink examples first to understand Apache Flink.
I just run some kafka with flink examples. No examples working for me. I
am so sad right now.
I didn't get
Hi Hawin!
If you are creating code for such an output into different
files/partitions, it would be amazing if you could contribute this code to
Flink.
It seems like a very common use case, so this functionality will be useful
to other user as well!
Greetings,
Stephan
On Tue, Jun 23, 2015 at
Dear Hawin,
We do not have out of the box support for that, it is something you would
need to implement yourself in a custom SinkFunction.
Best,
Marton
On Mon, Jun 22, 2015 at 11:51 PM, Hawin Jiang hawin.ji...@gmail.com wrote:
Hi Marton
if we received a huge data from kafka and wrote to
Hi Marton
if we received a huge data from kafka and wrote to HDFS immediately. We
should use buffer timeout based on your URL
I am not sure you have flume experience. Flume can be configured buffer
size and partition as well.
What is the partition.
For example:
I want to write 1 minute buffer
Thanks Marton
I will use this code to implement my testing.
Best regards
Hawin
On Wed, Jun 10, 2015 at 1:30 AM, Márton Balassi balassi.mar...@gmail.com
wrote:
Dear Hawin,
You can pass a hdfs path to DataStream's and DataSet's writeAsText and
writeAsCsv methods.
I assume that you are
