Re: Kinesis connector classpath issue when running Flink 1.1-SNAPSHOT on YARN

Hi Aljoscha, I opened an issue here https://issues.apache.org/jira/browse/FLINK-4115 and submitted a pull request. I'm not sure if my fix is the best way to resolve this, or if it's better to just remove the verification checks completely. Thanks, Josh On Thu, Jun 23, 2016 at 9:41 AM, Aljoscha

Question regarding logging capabilities in flink

Hi, I was reading this link regarding logging in flink jobs (https://ci.apache.org/projects/flink/flink-docs-master/internals/logging.html) and with modifications to log4j files I am able to see logs in flink/log directory when I run job Standalone or on Yarn, but I was more interested in

Re: Localhost terminated

What do the log files say? Cheers, Till On Thu, Jun 23, 2016 at 4:46 PM, Debaditya Roy wrote: > Hello users, > > I have a problem with flink execution from cli. Each time I deploy from > the CLI, the job starts but then it terminates the localhost. > > > > *06/23/2016

Re: Iterate several kafka topics using the kafka connector

Thank you. It totally works as what we want which unions data streams. Best, Sendoh -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Iterate-several-kafka-topics-using-the-kafka-connector-tp7673p7680.html Sent from the Apache Flink User

Localhost terminated

Hello users, I have a problem with flink execution from cli. Each time I deploy from the CLI, the job starts but then it terminates the localhost. *06/23/2016 16:42:18Source: Custom Source -> Flat Map -> Sink: Unnamed(1/1) switched to SCHEDULED 06/23/2016 16:42:19Source: Custom Source

Scala/ReactiveMongo: type classes, macros and java.util.Serializable

Hello everybody, in the past days, I've written batch input/output formats for MongoDB. Initially, I've tried to use the non-blocking ReactiveMongo driver, which uses the type class pattern in Scala for the serialization logic. The library also exposes some pretty

Re: Stoppable Job And Web UI Questions

Hi Yan Chou Chen, 1. At the moment Flink sources have to implement a certain interface, StoppableFunction, to be stoppable. If they do, then you can stop them via the CLI or the web interface. This cannot be triggered from within a job. However, you have a far better way to

Re: Iterate several kafka topics using the kafka connector

It is possible to instantiate the FlinkKafkaConsumer with multiple topics [1]. Simply pass a list of topic names instead of a the name of a single topic. streams.add(env.addSource(new FlinkKafkaConsumer09<>(Arrays.asList("foo", "bar", "foobar"), new JSONSchema(), properties));

Re: Initialization of static variables

Ok, thanks for the explanation Till! On Thu, Jun 23, 2016 at 3:19 PM, Till Rohrmann wrote: > Yes this is normal Flink behaviour. The reason is that static variables > are not transferred to the cluster. What happens instead when you first > load the class on the cluster is

Re: Initialization of static variables

Yes this is normal Flink behaviour. The reason is that static variables are not transferred to the cluster. What happens instead when you first load the class on the cluster is that the static variables are created and possible class initializer are executed. That is also the reason why your

Iterate several kafka topics using the kafka connector

Hi Flink developers, Can I ask how could we iterate several Kafka topics using the Kafka connector? Our idea is like the following example: List> streams = new ArrayList<>(); // Iterate kafka topics Iterator topicIter = topicList.iterator(); while

Initialization of static variables

Hi all, I've a Flink job that initialize a static Map in the main program, before starting any Flink transformation. If I run the job locally that variable is not empty, running the job on the cluster reset that variable..is it a bug or am I doing something wrong? It only works if I initialize

Re: State key serializer has not been configured in the config.

We should adjust the error message to contain the keyed stream thingy. On 23.06.2016 10:11, Till Rohrmann wrote: Hi Jacob, the `ListState` abstraction is a state which we call partitioned/key-value state. As such, it is only possible to use it with a keyed stream. This means that you have to

Re: Kinesis connector classpath issue when running Flink 1.1-SNAPSHOT on YARN

Hi Josh, do you maybe want to open an issue for that and contribute your fix for that? Cheers, Aljoscha On Fri, 17 Jun 2016 at 17:49 Josh wrote: > Hi Aljoscha, > > Thanks! It looks like you're right. I've ran it with the FsStateBackend > and everything works fine. > > I've

Re: State key serializer has not been configured in the config.

Hi Jacob, the `ListState` abstraction is a state which we call partitioned/key-value state. As such, it is only possible to use it with a keyed stream. This means that you have to call `keyBy` after the `connect` API call. Cheers, Till On Wed, Jun 22, 2016 at 9:17 PM, Jacob Bay Larsen