Re: Flink to ingest from Kafka to HDFS?

Hi! Sorry, I won't be able to implement this soon. I just shared my ideas on this. Greets. Rico. > Am 25.08.2015 um 17:52 schrieb Stephan Ewen : > > Hi Rico! > > Can you give us an update on your status here? We actually need something > like this as well (and pretty urgent), so we would

Re: Application-specific loggers configuration

Hi Gwenhaël, are you using the one-yarn-cluster-per-job mode of Flink? I.e., you are starting your Flink job with (from the doc): flink run -m yarn-cluster -yn 4 -yjm 1024 -ytm 4096 ./examples/flink-java-examples-0.10-SNAPSHOT-WordCount.jar If you are, then this is almost possible on the current

Re: Source & job parallelism

Hi Arnaud, did you try: > Env.setSource(mySource).setParrellelism(1).map(mymapper).setParallelism(10) If this does not work, it might be that Flink chains the mapper to the source which implies to use the same parallelism (and the producer dictates this dop value). Using a rebalance() in betwee

Source & job parallelism

Hi, I have a streaming source that extends RichParallelSourceFunction, but for some reason I don’t want parallelism at the source level, so I use : Env.setSource(mySource).setParrellelism(1).map(mymapper) I do want parallelism at the mapper level, because it’s a long task, and I would like the

Re: Flink to ingest from Kafka to HDFS?

Hi Rico! Can you give us an update on your status here? We actually need something like this as well (and pretty urgent), so we would jump in and implement this, unless you have something already. Stephan On Thu, Aug 20, 2015 at 12:13 PM, Stephan Ewen wrote: > BTW: This is becoming a dev disc

[ANNOUNCE] Flink Forward 2015 program is online

Hi everyone, Just a shoutout that we have posted the program of Flink Forward 2015 here: http://flink-forward.org/?post_type=day You can expect few changes here and there, but the majority of the talks is in. Thanks again to the speakers and the reviewers! If you have not registered yet, now is

Re: Broadcasting sets in Flink Streaming

Ok, I'll try that. Thanks a lot! On Tue, Aug 25, 2015 at 4:19 PM, Stephan Ewen wrote: > You can do something very similar like broadcast sets like this: > > Use a Co-Map function and connect your main data set regularly ("forward" > partitioning) to one input and your broadcast set via "broadcas

Re: Broadcasting sets in Flink Streaming

You can do something very similar like broadcast sets like this: Use a Co-Map function and connect your main data set regularly ("forward" partitioning) to one input and your broadcast set via "broadcast" to the other input. You can then retrieve the data in the two map functions separately. This

Application-specific loggers configuration

Hi, We're developing the first of (we hope) many flink streaming app. We'd like to package the logging configuration (log4j) together with the jar. Meaning, different application will probably have different logging configuration (ex: to different logstash ports) ... Is there a way to "overrid

Re: Broadcasting sets in Flink Streaming

Hi Tamara, I think this is not officially supported by Flink yet. However, I think that Gyula had once an example where he did something comparable. Maybe he can chime in here. Cheers, Till On Tue, Aug 25, 2015 at 11:15 AM, Tamara Mendt wrote: > Hello, > > I have been trying to use the functio

Broadcasting sets in Flink Streaming

Hello, I have been trying to use the function withBroadcastSet on a SingleOutputStreamOperator (map) the same way I would on a MapOperator for a DataSet. From what I see, this cannot be done. I wonder if there is some way to broadcast a DataSet to the tasks that are performing transformations on a