Re: REST Endpoint for Triggering Savepoint

No, this is not currently supported, but we plan to include this in 1.5 . On 07.11.2017 20:01, vijayakumar palaniappan wrote: Hi, Is there a REST endpoint for triggering savepoint without cancelling the job? -- Thanks, -Vijay

REST Endpoint for Triggering Savepoint

Hi, Is there a REST endpoint for triggering savepoint without cancelling the job? -- Thanks, -Vijay

AvroParquetWriter may cause task managers to get lost

Hi all, We recently implemented a feature in our streaming flink job in which we have a AvroParquetWriter which we build every time the overridden “write” method from org.apache.flink.streaming.connectors.fs.Writer gets called. We had to do this because the schema of each record is potentially

Re: Kafka Consumer fetch-size/rate and Producer queue timeout

Thanks Fabian. I am seeing thia consistently and can definitely use some help. I have plenty of graphana views I can share if that helps :) Sent from Yahoo Mail on Android On Tue, Nov 7, 2017 at 3:54 AM, Fabian Hueske wrote: Hi Ashish, Gordon (in CC) might be able to

Re: Flink memory usage

I’ve used the following simple script to capture Flink metrics by running: python -u ./statsd_server.py 9020 > statsd_server.log >>> flink-conf.yaml metrics.reporters: statsd_reporter metrics.reporter.statsd_reporter.class: org.apache.flink.metrics.statsd.StatsDReporter

Re: Job Manager Configuration

Hi Regina, the user code is uploaded once to the `JobManager` and then downloaded from each `TaskManager` once when it first receives the command to execute the first task of your job. As Chesnay said there is no fundamental limitation to the size of the Flink job. However, it might be the case

Re: FlinkCEP behaviour with time constraints not as expected

Hi Federico, For your given input and pattern there should (and there are) only two timeouted patterns: 5> Left(Map(start -> List(Event(100,2017-11-05T03:56:02 5> Left(Map(start -> List(Event(100,2017-11-05T06:00:02 It is because in your patterns say the next event after events with

Re: ExecutionGraph not serializable

Hi XiangWei, it is actually not intended to get access to the ExecutionGraph, because it is a runtime component which does not make much sense to exist outside of the JobManager. The RequestJob message is only a hack to make the ExecutionGraph accessible to another actor running in the same

Call for responses: Apache Flink user survey 2017

Hi everyone, data Artisans is running a second annual Apache Flink user survey [1] in order to understand Flink usage and the needs of the community. This survey will help to shape the Flink roadmap and make Flink the best that it can be for users. We'll publish a report with a summary of

Getting java.lang.ClassNotFoundException: for protobuf generated class

Hi, I am using flink 2.1.0 version and protobuf-java 2.6.1 version. I am getting below exception for protobuf generated class. I have included jar which is having that class. Can you please help me to check it. org.apache.beam.sdk.util.UserCodeException:

Re: Flink memory leak

I agree with Ufuk, it would be helpful to know what stateful operations are in the jobs (including windowing). > On 7. Nov 2017, at 14:53, Ufuk Celebi wrote: > > Do you use any windowing? If yes, could you please share that code? If > there is no stateful operation at all,

Re: Flink memory leak

Do you use any windowing? If yes, could you please share that code? If there is no stateful operation at all, it's strange where the list state instances are coming from. On Tue, Nov 7, 2017 at 2:35 PM, ebru wrote: > Hi Ufuk, > > We don’t explicitly define any

Re: ExecutionGraph not serializable

hi Till, Sorry,I've made a mistake,i used *StandaloneClusterClient*#*getJobManagerGateway *to get *ActorGateway *to communicate with *JobManager *instead of using *JobMasterGateway*. Below is the code i executed for getting ExecuteGraph of a Job. val flinkConfig = new Configuration()

Re: Flink memory leak

Hi Ufuk, We don’t explicitly define any state descriptor. We only use map and filters operator. We thought that gc handle clearing the flink’s internal states. So how can we manage the memory if it is always increasing? - Ebru > On 7 Nov 2017, at 16:23, Ufuk Celebi wrote: >

Re: MODERATE for d...@flink.apache.org

Have you tried removing the "cygdrive" portion from the path? Something along the lines of “state.backend.fs.checkpointdir: file:///Y:/flink-checkpoint-dir state.checkpoints.dir: Y:/flink-checkpoints state.backend: filesystem high-availability.storageDir: file:///Y:/flink-recovery " On

Re: MODERATE for d...@flink.apache.org

Dear Ufuk, Sorry, I still can’t get that work, I have double checked all the nodes can access the mapped drive Y. Here is my state backend configuration, and it writes files to local C drive. “state.backend.fs.checkpointdir: file:///cygdrive/Y/flink-checkpoint-dir state.checkpoints.dir:

Re: Flink send checkpointing message in IT

hmm. While there is /technically/ no guarantee that notifyCheckpointComplete is called, it virtually always is, especially in local setups. Is it possible for you to share more code (or all of it)? (you can also send it to me directly) On 07.11.2017 11:58, Rinat wrote: Yes, but

Re: Flink send checkpointing message in IT

Yes, but notifyCheckpointComplete callback doesn’t called on await completion, I do the same, as in specified test template : ActorGateway jobManager = (ActorGateway) Await.result(cluster.leaderGateway().future(), DEADLINE.timeLeft()); Future savepointResultFuture =

Re: Flink send checkpointing message in IT

Do you verify that savepointResult is a JobManagerMessages.TriggerSavepointSuccess? It could also be JobManagerMessages.TriggerSavepointFailure. (instanceof check) On 02.11.2017 19:11, Rinat wrote: Chesnay, thanks for your reply, it was very helpful, but I took logic from this test template

Re: MODERATE for d...@flink.apache.org

Hey Jordan, yeah, that should just work. Check out the state backend configuration here: https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/state/state_backends.html – Ufuk On Tue, Nov 7, 2017 at 11:44 AM, Jordan Kuan wrote: > Dear Ufuk, > > Thank you for

Re: MODERATE for d...@flink.apache.org

Dear Ufuk, Thank you for your reply. All the cluster Flink servers are able to access network drive, and it mapped as drive Y in all nodes. Do I need to provide more information? Thanks, Jordan > On 7 Nov 2017, at 6:36 PM, Ufuk Celebi wrote: > > As answered by David on SO,

Re: MODERATE for d...@flink.apache.org

As answered by David on SO, the files need to be accessible by all nodes. In your setup this seems not to be the case, therefore it won't work. You need a distributed file system (e.g. NFS or HDFS) or object store (e.g. S3) that is accessible from all nodes. – Ufuk On Tue, Nov 7, 2017 at 3:34

Re: Flink memory leak

Hey Ebru, let me pull in Aljoscha (CC'd) who might have an idea what's causing this. Since multiple jobs are running, it will be hard to understand to which job the state descriptors from the heap snapshot belong to. - Is it possible to isolate the problem and reproduce the behaviour with only a

Re: FlinkCEP behaviour with time constraints not as expected

Hey Frederico, let me pull in Dawid (cc'd) who works on CEP. He can probably clarify the expected behaviour here. Best, Ufuk On Mon, Nov 6, 2017 at 12:06 PM, Federico D'Ambrosio wrote: > Hi everyone, > > I wanted to ask if FlinkCEP in the following scenario is

Flink memory leak

Hi, We are using Flink 1.3.1 in production, we have one job manager and 3 task managers in standalone mode. Recently, we've noticed that we have memory related problems. We use docker container to serve Flink cluster. We have 300 slots and 20 jobs are running with parallelism of 10. Also the

Re: ExecutionGraph not serializable

Hi XiangWei, how do you use the JobMasterGateway with the actor message RequestJob? The JobMasterGateway is a Java interface and does not represent an ActorCell to which you can send actor messages. Instead you should call JobMasterGateway#requestArchivedExecutionGraph. Cheers, Till ​ On Tue,

Re: ExecutionGraph not serializable

Hi XiangWei, I don't think this is a public interface, but Till (in CC) might know better. Best, Fabian 2017-11-06 3:27 GMT+01:00 XiangWei Huang : > Hi Flink users, > Flink Jobmanager throw a NotSerializableException when i used > JobMasterGateway to get ExecutionGraph