How to get the ActorSystem in custom operator

hi?? Since I want to use akka to sync some status in my custom operator??Is there any way to get the ActorSystem in org.apache.flink.streaming.api.operators.AbstractStreamOperator or get the address list of the taskmanager to build the ActorSystem?? thanks

Re: how get job id which job run slot

Hi lining, Yes, currently you can't get slot information via the "/taskmanagers/:taskmanagerid" rest API. In addition, please ask questions in the user mailing list. The dev mailing list mainly discusses information related to Flink development. Thanks, vino. lining jing 于2018年11月9日周五

Re: Queryable state when key is UUID - getting Kyro Exception

Yeah, it IS using Kryo serializer. Jayant Ameta On Wed, Nov 7, 2018 at 9:57 PM Till Rohrmann wrote: > Hi Jayant, could you check that the UUID key on the TM is actually > serialized using a Kryo serializer? You can do this by setting a breakpoint > in the constructor of the

The heartbeat of TaskManager with id ... timed out.

I am running Flink 1.7 on K8S. I am not sure how to debug this issue. I turned on debug on JM/TM. I am not sure this part is related or not. How could an Actor suddenly disappear? = 2018-11-09 04:47:19,480 DEBUG org.apache.flink.runtime.rest.handler.legacy.metrics.MetricFetcher - Query

Re: Always trigger calculation of a tumble window in Flink SQL

I am able to write a single operator as you suggested, thank you. And then I saw ContinuousProcessingTimeTrigger from flink source code, it looks like it's something that I am looking for, if there is a way that I can customize the SQL "TUMBLE" window to use this trigger instead of

Re: akka timeout exception

Thanks for the reply Dawid. The Flink jobs are deployed in Yarn cluster. I am seeing the error in Job Manager log for some jobs too frequently. I'm using Flink 1.4.2. I'm running only Streaming Jobs. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

How can I configure logback for TaskManager in LocalStreamEnvironment

I'm using Flink LocalStreamEnvironment when developing job. When I use the logback for logging, and I found that logback.xml only work for Jobmanager but not work for TaskManager. How can I configure the log layout and level for TaskManager in LocalStreamEnvironment. Thanks! PB

Re: ProcessFunction's Event Timer not firing

Hi Fritz, Watermarks are merged on stream shuffles. If one of the input's watermark not progressing, they will not advance the event time at the operators. I think you should decrease the parallelism of source and make sure there are data in each of your source partition. Note that the Kafka

Manually clean SQL keyed state

I have a scenario in which i do a non-windowed group by using SQL. something like "Select count(*) as events, shouldTrigger(..) as shouldTrigger from source group by sessionId" i'm then converting to a retracted stream, filtering by "add" messages, then further filtering by "shouldTrigger" field

Rich variant for Async IO in Scala

Hi, I see that the AsyncFunction for Scala does not seem to have a rich variant like the Java one. Is there a particular reason for this? Is there any workaround? Thanks! Bruno

Re: HA jobmanagers redirect to ip address of leader instead of hostname

Hi Till, Thanks for your reply. We are running version 1.5.4. We can't upgrade to 1.6.x because we are using Apache Beam which doesn't support 1.6.x yet. I have also made a Jira issue about this: https://issues.apache.org/jira/projects/FLINK/issues/FLINK-10748 Best regards, Jeroen Steggink

Re: Understanding checkpoint behavior

Hi, > On 6 Nov 2018, at 18:22, PranjalChauhan wrote: > > Thank you for your response Piotr. I plan to upgrade to Flink 1.5.x early > next year. > > Two follow-up questions for now. > > 1. > " When operator snapshots are taken, there are two parts: the synchronous > and the asynchronous

Re: akka timeout exception

Hi， I got the same exception when running in flink cluster. The settings is below: flink version: 1.5.4 flink-conf.yaml: jobmanager.heap.mb: 102400 taskmanager.heap.mb: 102400 taskmanager.numberOfTaskSlots: 40 parallelism.default: 40 I have 5 task manager. My code just read hbase table data

Re: FlinkCEP, circular references and checkpointing failures

Thanks a lot for looking into this issue Stefan. Could you please let me know the issue ID once you open it? It'll help me understand the problem better, and also I could do a quick test in our environment once the issue is resolved. Thanks, Shailesh On Wed, Nov 7, 2018, 10:46 PM Till Rohrmann

Multiple operators to the same sink

Hi Guys! I'm designing a topology where multiple operators should forward the messages to the same sink. For example I have Operator A,B,C,D,E. I want A,B,C to forward to Sink1 and D, E to forward to Sink2. My options are 1. Union A, B and C. then add Sink1 to them. Similarly for D and E.

Re: FlinkCEP, circular references and checkpointing failures

Sure, it is already merged as FLINK-10816. Best, Stefan > On 8. Nov 2018, at 11:53, Shailesh Jain wrote: > > Thanks a lot for looking into this issue Stefan. > > Could you please let me know the issue ID once you open it? It'll help me > understand the problem better, and also I could do a

Task Manager allocation issue when upgrading 1.6.0 to 1.6.2

I'm running a YARN cluster of 8 * 4 core instances = 32 cores, with a configuration of 3 slots per TM. The cluster is dedicated to a single job that runs at full capacity in "FLIP6" mode. So in this cluster, the parallelism is 21 (7 TMs * 3, one container dedicated for Job Manager). When I run

Re: java.io.IOException: NSS is already initialized

Thanks, any insight/help here is appreciated. On Thu, Nov 8, 2018 at 4:38 AM Dawid Wysakowicz wrote: > Hi Hao, > > I am not sure, what might be wrong, but I've cc'ed Gary and Kostas who > were recently working with S3, maybe they will have some ideas. > > Best, > > Dawid > On 03/11/2018 03:09,

RE: Stopping a streaming app from its own code : behaviour change from 1.3 to 1.6

1.FLINK-10832 Created (with heavy difficulties as typing java code in a jira description was an awful experience ☺) De : LINZ, Arnaud Envoyé : mercredi 7 novembre 2018 11:43 À : 'user' Objet : RE: Stopping a streaming app from its own code

Re: Always trigger calculation of a tumble window in Flink SQL

Hi Piotr, Thank you for your explanation. I basically understand your meaning, as far as my understanding, we can write a custom window assigner and custom trigger, and we can register the timer when the window process elements. But How can we register a timer when no elements received during

Re: Always trigger calculation of a tumble window in Flink SQL

Hi Piotr, Thank you for your explanation. I basically understand your meaning, as far as my understanding, we can write a custom window assigner and custom trigger, and we can register the timer when the window process elements. But How can we register a timer when no elements received during

Re: FlinkKafkaProducer and Confluent Schema Registry

Dawid, Is there a projected date to deliver ConfluentRegistryAvroSerializationSchema ? thank you, Olga From: Dawid Wysakowicz Sent: Monday, October 22, 2018 10:40 AM To: trebl...@hotmail.com Cc: user Subject: Re: FlinkKafkaProducer and Confluent Schema Registry

Re: Flink cluster security conf.: keberos.keytab add to run yarn-cluster

Hi, Wouldn't `-yD` option do the trick? I use it to override the kerberos configuration for different users very often. Best, Paul Lam > 在 2018年11月8日，17:33，Dawid Wysakowicz 写道： > > Hi Marke, > > AFAIK Shuyi is right, there is no such option so far. Maybe you could do > though is to extend

Re: Always trigger calculation of a tumble window in Flink SQL

Hi Piotr, Thank you for your explanation. I basically understand your meaning, as far as my understanding, we can write a custom window assigner and custom trigger, and we can register the timer when the window process elements. But How can we register a timer when no elements received during a

Re: Always trigger calculation of a tumble window in Flink SQL

Hi Piotr, Thank you for your explanation. I basically understand your meaning, as far as my understanding, we can write a custom window assigner and custom trigger, and we can register the timer when the window process elements. But How can we register a timer when no elements received during a

Re: Always trigger calculation of a tumble window in Flink SQL

Re-adding user mailing list to CC Hi, > I basically understand your meaning, as far as my understanding, we can write > a custom window assigner and custom trigger, and we can register the timer > when the window process elements. No I was actually suggesting to write your own operator to

Re: flink job restarts when flink cluster restarts?

Thanks! If I have a cluster more than one node (standalone or YRAN), can I stop and start any single node among them and keep the job running? Best regards/祝好, Chang Liu 刘畅 > On 7 Nov 2018, at 16:17, 秦超峰 <18637156...@163.com> wrote: > > the second > > > > > 秦超峰 >

Re: flink job restarts when flink cluster restarts?

Or to say, how can I keep the jobs for system patching, server restart, etc. Is it related to Standalone vs YARN? Or is it related to whether to use Zookeeper? Many thanks! Best regards/祝好, Chang Liu 刘畅 > On 8 Nov 2018, at 13:38, Chang Liu wrote: > > Thanks! > > If I have a cluster more

ProcessFunction's Event Timer not firing

Hi All, I noticed if one of the slot's watermark not progressing, its impacting all slots processFunction timer and no timer are not firing. In my example, I have Source parallelism set to 8 and Kafka partition is 4. The next operator is processFunction with parallelism of 8 + event timer. I

Re: FlinkKafkaProducer and Confluent Schema Registry

Hi Olga, The only thing I can tell is that it definitely won't make it to 1.7 release. The earliest possible is 1.8 then, which is scheduled for the beginning of next year. Best, Dawid On 08/11/2018 00:48, Olga Luganska wrote: > Dawid, > > Is there a projected date to > deliver 

Re: Flink weird checkpointing behaviour

Hi, I think it is definitely worth checking the alignment time as Yun Tang suggested. There were some changes in the network stack that could influence this behavior between those version. I've also added Stefan as cc, who might have more ideas what would be worth checking. Best, Dawid On

Re: "org.apache.flink.client.program.ProgramInvocationException: Unknown I/O error while extracting contained jar files

Hi, Could you post the full stacktrace of the exception? Best, Dawid On 05/11/2018 09:19, wangziyu wrote: > Hi, > I use monitor Restful api ，“/jars/{jars}/run” to test my environment.The > exception is happend. > I did exactly that： > 1.I use “/jars/upload” to upload my jar. > 2.I

Re: sys.exist(1) led to standalonesession daemon closed

Hi Tony, I think your reasoning is correct that this is because of the fact that rest server runs in the same process as standalonesession. It's hard to say if it is an expected behavior or not, but there is really not much we can do about it. User code can actually call System.exit from any

Re: java.io.IOException: NSS is already initialized

Hi Hao, I am not sure, what might be wrong, but I've cc'ed Gary and Kostas who were recently working with S3, maybe they will have some ideas. Best, Dawid On 03/11/2018 03:09, Hao Sun wrote: > Same environment, new error. > I can run the same docker image with my local Mac, but on K8S, this >

Re: Question about slot and yarn vcores

Hi, It is not as easy. To understand it well I would recommend going through the docs[1]. In short slots do not equal thread/core it is just an abstraction over a share of resources. For you setup: It is true there will be 3 jvms each will be assigned 2 vcores ( by default it is equal to slot

Re: Starting a seperate Java process within a Flink cluster

Hi, I am afraid that would be extremely hard what you are trying to do as in a cluster setup not all dependencies are taken from the taskmanager classpath, actually the user code classes are loaded dynamically, therefore they cannot be accessed in your new process which does not have access to

Re: Job manager UI improvement

Hi Michael, There are no metrics for actual state size so far, but Yun Tang's suggestion is the best what you can do right now. You can also refer to a similar previous thread in the ML[1] I also add Aljosha to cc, who might know if there was any progress on this topic since then. Best,

Re: TaskManagers cannot contact JobManager in Kubernetes when JobManager HA is enabled

Hi John, Glad you resolved the issue. Also thanks for sharing the solution with ML! Best, Dawid On 01/11/2018 16:22, John Stone wrote: > I've managed to resolve the issue. With HA enabled, you will see this > message in the logs: > > 2018-11-01 13:38:52,467 INFO >

Re: Question about serialization and performance

Hi Michael, Things I could suggest are: * first of all KryoSerialization is sort of a fallback serialization that is used if there is no better suited serialization that can be used, as you said you are using some complex JSONObjects I would recommend writing your own

Re: akka timeout exception

Hi, Could you provide us with some more information? Which version of flink are you running? In which cluster setup? When does this exception occur? This exception says that request for status overview (no of taskmanagers, slots info etc.) failed. Best, Dawid On 31/10/2018 20:05, Anil wrote: >

Re: Flink cluster security conf.: keberos.keytab add to run yarn-cluster

Hi Marke, AFAIK Shuyi is right, there is no such option so far. Maybe you could do though is to extend the "flink" script to substitute those parameters in the file on each run, but I think it is a common practice to run flink jobs on yarn from a single service user. On 31/10/2018 19:52, Shuyi

Re: flink-1.6.1 :: job deployment :: detached mode

Hi Mike, could you also send me the YarnJobClusterEntrypoint logs. Thanks! Cheers, Till On Wed, Nov 7, 2018 at 9:27 PM Mikhail Pryakhin wrote: > Hi Till, > Thank you for your reply. > Yes, I’ve upgraded to the latest Flink-1.6.2 and the problem is still > there, please find the log file

Re: Flink 1.6, User Interface response time

Hi Oleksandr, Have you checked the jobmanager logs to see if there are any exceptions? What is the response code for request when it doesn't load? Best, Dawid On 31/10/2018 16:49, Oleksandr Nitavskyi wrote: > > Hello! > >   > > We are migrating the the last 1.6.2 version and all the jobs seem