[no subject]

entrypoint for executing job in task manager

Here is my understanding of how job submission works in Flink. When submitting a job to job manager via REST API, we provide a entry class. Job manager then evaluate job graph and ship serialized operators to task manager. Task manager then open operators and run tasks. My app would typically

Re: Flink upgrade compatibility table

Hi Kien, Thanks for the feedback. I wasn't certain regarding compatibility between jars. I did version bump the flink libraries and the application did start. Just curious if the previous jar still worked without upgrading. Regarding the savepoint table. Someone should probably add 1.4

Flink upgrade compatibility table

I recently tried to launch our application 1.3 jars against a 1.4 cluster. I got a java.lang.NoClassDefFoundError: org/apache/flink/streaming/api/checkpoint/CheckpointedRestoring when I tried to run our 1.3 flink application against 1.4 . Then I googled around and didn't see a mention of 1.4 in

Re: Restroring from a SP

Hi Vishal, AFAIK, intermittent restore failures from savepoints should not be expected. Do you still have the logs from the failed restore attempts? What exceptions were the restores failing on? We would need to take a look at the logs to figure what may be going on. Best, Gordon -- Sent

Re: Apache Flink - Difference between operator and function

Hi Mans, What's the difference between an operator and a function ?  An operator in Flink needs to handle processing of watermarks, records, and checkpointing of the operator state. To implement one, you need to extend the AbstractStreamOperator base class. It is considered a very low-level API

Re: A question about Triggers

And that further begs the question.. how performant is Timer Service. I tried to peruse through the architecture behind it but cold not find a definite clue. Is it a Scheduled Service and if yes how many threads etc... On Wed, Dec 20, 2017 at 4:25 PM, Vishal Santoshi

Restroring from a SP

Are intermittent failures to restore from a SP, in the case of flink offset a know issue ? I had more than one instance where the offsets were not restored, but a retry ( in one case I it succeeded like the 4th restore attempt ) . I am on 1.3.2.

Re: A question about Triggers

Makes sense. Did a first stab at Using ProcessFunction. The TimeService exposed by the Context does not have remove timer. Is it primarily b'coz A Priority Queue is the storage ad remove from a PriorityQueue is expensive ? Trigger Context does expose another version that has removal abilities so

Apache Flink - Difference between operator and function

Hi: I am reading the documentation on working with state (https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/state.html) and it states that : All datastream functions can use managed state, but the raw state interfaces can only be used when implementing operators.

Re: Pending parquet file with Bucking Sink

Hi Vipul, Thanks for the information. Yes, I do have checkpointing enabled with 10 millisecs. I think the issue here is that the stream ended before the checkpoint reached. This is a testing code that the DataStream only have 5 events then it ended. Once the stream ended, the checkpoint is

Metric reporters with non-static ports

The prometheus metric reporter allows for a specification of a port range; is there a way I can find out which actual port it found to bind to? Also, there doesn’t seem to be a way to reserve an extra port for task managers in mesos to assign to a metric reporter, is that a roadmap item? I’m

Re: Can not resolve org.apache.hadoop.fs.Path in 1.4.0

Libraries such as CEP or Table API should have the "compile" scope and should be in the both the fat and non-fat jar. The non-fat jar should contain everything that is not in flink-dist or your lib directory. Regards, Timo Am 12/20/17 um 3:07 PM schrieb shashank agarwal: Hi, In that

Re: Can not resolve org.apache.hadoop.fs.Path in 1.4.0

Hi, In that case, it won't find the dependencies. Cause I have other dependencies also and what about CEP etc. cause that is not part of flink-dist. Best Shashank ‌ On Wed, Dec 20, 2017 at 3:16 PM, Aljoscha Krettek wrote: > Hi, > > That jar file looks like it has too

Re: flink jobmanager HA zookeeper leadership election - RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested.

Hi Colin, the log looks as if the Flink JobManager receives a SIGTERM signal and shuts down due to that. This is nothing which should be triggered by Flink's leader election. Could you check whether this signal might be created by another process in your environment or if the container supervisor

Re: Static Variables

Hi, I think the cause is very likely a race condition between the tasks checking and setting the static value, because tasks run in different threads. You could try to use an Atomic reference or synchronization for setting the state variable’s value. Best, Stefan > Am 20.12.2017 um 00:29

Re: NullPointerException with Avro Serializer

It turn out that our flink branch is out-of-date. Sorry for all the noise. :) Regards, Kien ⁣Sent from TypeApp ​ On Dec 20, 2017, 16:42, at 16:42, Kien Truong wrote: >Upon further investigation, we found out that the reason: > >* The cluster was started on YARN with

Re: Can not resolve org.apache.hadoop.fs.Path in 1.4.0

Hi, That jar file looks like it has too much stuff in there that shouldn't be there. This can explain the errors you seeing because of classloading conflicts. Could you try not building a fat-jar and have only your code in your jar? Best, Aljoscha > On 20. Dec 2017, at 10:15, shashank agarwal

Re: NullPointerException with Avro Serializer

Upon further investigation, we found out that the reason: * The cluster was started on YARN with the hadoop classpath, which includes Avro. Therefore, Avro's SpecificRecord class was loaded using sun.misc.Launcher$AppClassLoader * Our LteSession class was submitted with the application jar,

Re: Can not resolve org.apache.hadoop.fs.Path in 1.4.0

One more thing when i submit the job ir start yarn session it prints following logs : Using the result of 'hadoop classpath' to augment the Hadoop classpath:

Re: Add new slave to running cluster?

Hey Jinhua, - The `slaves` file is only relevant for the startup scripts. You can add as many task managers as you like by starting them manually. - You can check the logs of the JobManager or its web UI (jobmanager-host:8081) to see how many TMs have registered. - The registration time out looks

Re: Can not resolve org.apache.hadoop.fs.Path in 1.4.0

Hi, Could you please list what exactly is in your submitted jar file, for example using "jar tf my-jar-file.jar"? And also what files exactly are in your Flink lib directory. Best, Aljoscha > On 19. Dec 2017, at 20:10, shashank agarwal wrote: > > Hi Timo, > > I am

Re: How to apply patterns from a source onto another datastream?

Would it be possible to get the same result using windows? Jayant Ameta On Tue, Dec 19, 2017 at 3:23 PM, Dawid Wysakowicz < wysakowicz.da...@gmail.com> wrote: > It is not possible at this moment. FlinkCEP can handle only one Pattern > applied statically. There is a JIRA ticket for that: >