Issue with CEP library

Hi, I've been only working with flink for the past 2 weeks on a project and am trying using the CEP library on sensor data. I am using flink version 1.3.2. Flink has a kafka source. I am using KafkaSource9. I am running Flink on a 3 node AWS cluster with 8G of RAM running Ubuntu 16.04. From the

Re: Add custom configuration files to TMs classpath on YARN

What we internally did is to inherit from AbstractYarnClusterDescriptor and to customize from there. It's not too difficult but it's nice to see it can be taken care of by AbstractYarnCusterDescriptor. ~Haohui On Wed, Sep 27, 2017 at 9:36 AM Mikhail Pryakhin wrote: > Hi

Re: Flink Application Jar file on Docker container

There was also a talk about containerization at Flink Forward that touches on some of your questions. https://www.youtube.com/watch?v=w721NI-mtAA=2s=PLDX4T_cnKjD0JeULl1X6iTn7VIkDeYX_X=33 Eron On Wed, Sep 27, 2017 at 9:12 AM, Stefan Richter wrote: > Hi, > > from

Programmatic configuration

Hello, I’m running a single Flink Job Manager with a Task Manager in Docker containers with Java 8. They are remotely located (flink.example.com ). I’m submitting a job from my desktop and passing the job to the Job Manager with -m flink.example.com:6123

Re: Add custom configuration files to TMs classpath on YARN

Hi Robert, Thanks for your reply! >I believe you should be able to load non-class files through the classloader >as well. Could you please clarify what you mean by this? >Did you see any code that excludes non-class files? No I didn't, but I did see the following code here [1]: if

Re: CustomPartitioner that simulates ForwardPartitioner and watermarks

Hi Kostas, Yes, you have summarized well. I want to only forward the data to the next local operator, but broadcast the watermark through the cluster. - I can’t set parallelism of taskB to 1. The stream is too big for that. Also, the data is ordered at each partition. I don’t want to change

Re: Flink Application Jar file on Docker container

Hi, from the top of my head, I cannot see why this should not be possible, task managers just need to be able to connect to their job manager. Unfortunately, I cannot give a real guarantee here because I am not that deeply involved in this aspect of Flink. Best, Stefan > Am 27.09.2017 um

Re: Add custom configuration files to TMs classpath on YARN

Hi Mike, For using the DistributedCache approach, you need to have HDFS or another distributed FS available to distribute the files. I would actually like to understand why you said " then this file is copied to the yarn cluster and added to JVM class [...] but is ignored by TM JVM as it is

Re: Add custom configuration files to TMs classpath on YARN

Hi Nico, Thanks a lot for you help, but unfortunately, the workaround you suggested doesn't work for me. I tried to leverage the StreamExecutionEnvironment#registerCachedFile method but failed because this instance is created when the application master has already been started therefore the

Re: Flink Application Jar file on Docker container

Hi Stefan, I have a question in my mid out of curiosity Is it possible to run flink application within docker container by using flink cluster set up on host? Rahul Raj On 26 September 2017 at 17:29, Stefan Richter wrote: > Hi, > > if I correctly understood

Re: Using latency markers

Any follow-up on this? Jira? PR? On Wed, Sep 13, 2017 at 11:30 AM, Tzu-Li (Gordon) Tai wrote: > Hi Aitozi, > > Yes, I think we haven’t really pin-pointed out the actual cause of the > problem, but if you have a fix for that and can provide a PR we can > definitely look at

Re: got Warn message - "the expected leader session ID did not equal the received leader session ID " when using LocalFlinkMiniCluster to interpret scala code

Hi Till, I’ve found that a StandaloneMiniCluster doesn’t startup web fronted when it is running.so,how can i cancel a running job on it with restful method. Cheers, Till > 在 2017年9月20日，15:43，Till Rohrmann 写道： > > Hi XiangWei, > > programmatically there

Re: StreamCorruptedException

Hi Sridhar, From looking at your code: 1) The “KafkaDataSource” is a custom source that you implemented? Does this source buffer anything? 2) The getStreamSource2 seems to return again a "new KafkaDataSource”. Can this be a problem? 3) You are working on processing time and you are simply

Re: Stream Task seems to be blocked after checkpoint timeout

Hi Tony, are your checkpoints typically close to the timeout boundary? From what I see, writing the checkpoint is relatively fast but the time from the checkpoint trigger to execution seems very long. This is typically the case if your job has a lot of backpressure and therefore the checkpoint

Re: Stream Task seems to be blocked after checkpoint timeout

Hi Stefan, It seems that I found something strange from JM's log. It had happened more than once before, but all subtasks would finish their checkpoint attempts in the end. 2017-09-26 01:23:28,690 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 1140 @

Re: Building scala examples

Hi Michael, yes, it seems that the self-contained jars only contain the Java examples. You may also follow the quickstart [1] to get started writing Flink streaming programs in Scala. Nico [1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/quickstart/ scala_api_quickstart.html

Re: Exception in BucketingSink when cancelling Flink job

Hi, I would speculate that the reason for this order is that we want to shutdown the tasks quickly by interrupting blocking calls in the event of failure, so that recover can begin as fast as possible. I am looping in Stephan who might give more details about this code. Best, Stefan > Am

Re: Stream Task seems to be blocked after checkpoint timeout

Hi, thanks for the information. Unfortunately, I have no immediate idea what the reason is from the given information. I think most helpful could be a thread dump, but also metrics on the operator operator level to figure out which part of the pipeline is the culprit. Best, Stefan > Am

Re: Cannot deploy Flink on YARN

Since you're running in a container, the question is whether the container where the JM is running can access the ZooKeeper at 10.200.0.6. > On 27. Sep 2017, at 04:31, Sridhar Chellappa wrote: > > Emily, > > I did not get chance to capture the logs on the container.

Re: How to user ParameterTool.fromPropertiesFile() to get resource file inside my jar

Hi Ron, This is the issue and it was already implemented for upcoming 1.3.3 and 1.4.0. https://issues.apache.org/jira/browse/FLINK-7630 Best, Aljoscha > On 26. Sep 2017, at 21:10, Ron Crocker wrote: > > What’s crazy