Job Manager becomes irresponsive if the size of the session cluster grows

Hello, We have deployed multiple Flink clusters on Kubernetess with 1 replica of Jobmanager and multiple of Taskmanager as per the requirement. Recently we are observing that on increasing the number of Taskmanagers for a cluster, the Jobmanager becomes irresponsive. It stops sending statsd

Re: Flink 的 log 文件夹下产生了 44G 日志

尴尬了，之前回复的邮件难道都是空白，Henry 你可以把报错信息放到哪个博客里面，然后再这里提供个链接，邮件看不到你的截图错误信息，你可以先根据错误信息定位问题所在，把错误的问题先解决掉，然后再来合理的配置重启策略。 Biao Liu 于2019年7月18日周四 上午1:15写道： > Hi Henry, > > 邮件列表貌似不能支持直接贴图，所以无法理解“里面不停的在产生 error > 信息，但是也没有看到具体哪里报错”是什么意思。尝试贴图到第三方，然后贴链接上来？不知道还有没有更好的办法 > > To zhisheng2018, 你为什么老回空邮件？ > > >

Re: Cannot access the data from Hive-Tables in Blink

Hi Yebgenya, This is caused by Hive version mismatch, you are either not using the right Hive version (double check your Hive version is supported by Blink), or not specifying the right version in yaml config (e.g. you use 2.3.4 but specify it as 1.2.1). Bowen On Tue, Jul 16, 2019 at 11:22 AM

Re:Re: Job leak in attached mode (batch scenario)

There should be no JIRA about the requirement. If you have a strong need for this feature, you can create one. In addition, you can also go to issues.apache.org and search with keywords to confirm whether there are the relevant JIRA. Best, Haibo At 2019-07-18 10:31:22, "qi luo" wrote:

????StreamingFileSink????BucketingSink????HDFS ????????

??StreamingFileSinkBucketingSinkHDFS?? 1in-process?? 2??0??

Re:yarn-session vs cluster per job for streaming jobs

Hi, Maxim For the concern talking on the first point： If HA and checkpointing are enabled, AM (the application master, that is the job manager you said) will be restarted by YARN after it dies, and then the dispatcher will try to restore all the previously running jobs correctly. Note that

Re: [DISCUSS] Create a Flink ecosystem website

Robert and Daryl, thanks for the great work, I tried the website and filed some issues on Github. Best, Congxian Robert Metzger 于2019年7月17日周三 下午11:28写道： > Hey all, > > Daryl and I have great news to share. We are about to finish adding the > basic features to the ecosystem page. > We are at a

ValueState in SessionWindows

I implemented a class， the class extends MergingWindowAssigner, In the trigger, I use ValueState, when windows is merged, I want to get ValueState in merged windows. now I can't get the ValueState.

[no subject]

Hello, I am trying to use HashMap In my window function of flink job. if the parallelism change, is this hashmap still a singleton? Shouldn’t I do something similar here? 发送自 Windows 10 版邮件应用

Re: Job leak in attached mode (batch scenario)

Thanks Haibo for the response! Is there any community issue or plan to implement heartbeat mechanism between Dispatcher and Client? If not, should I create one? Regards, Qi > On Jul 17, 2019, at 10:19 AM, Haibo Sun wrote: > > Hi, Qi > > As far as I know, there is no such mechanism now. To

Re: table toRetractStream missing last record and adding extra column (True)

yes even the delimiter can be replaced, have to test what happens if the data itself has a comma in it I need to test. table.toRetractStream(TypeInformation.of(classOf[Row])) .map(_._2.toString.replaceAll(",","~"))

Re: table toRetractStream missing last record and adding extra column (True)

Amazing all issues resolved in one go thanks Cheng , one issue though I can't write map.(_._2) to CSV looks like it doesn't support right now have to be TextFile. below is a full code if someone wants in Scala. Git Code is here:- https://github.com/kali786516/FlinkStreamAndSql package

Flink Zookeeper HA: FileNotFoundException blob - Jobmanager not starting up

Hello, I've got a problem with our flink cluster where the jobmanager is not starting up anymore, because it tries to download non existant (blob) file from the zookeeper storage dir. We're running flink 1.8.0 on a kubernetes cluster and use the google storage connector [1] to store checkpoints,

Re: Providing external files to flink classpath

Does the -yt option work for standalone cluster without dedicated resource manager ? So this property file is read by one of the dependencies inside my application as a file, so I can't really use Parameter tool to parse the config file. Thanks, Vishwas On Fri, Jun 28, 2019 at 11:08 PM Yun Tang

Re: Flink 的 log 文件夹下产生了 44G 日志

Hi Henry, 邮件列表貌似不能支持直接贴图，所以无法理解“里面不停的在产生 error 信息，但是也没有看到具体哪里报错”是什么意思。尝试贴图到第三方，然后贴链接上来？不知道还有没有更好的办法 To zhisheng2018, 你为什么老回空邮件？ zhisheng2...@gmail.com 于2019年7月18日周四 上午12:49写道： >

回复：Flink 的 log 文件夹下产生了 44G 日志

回复：flink web ui 添加 subtask index 信息

Flink and CDC

Hi to all, I'd like to know whether it exists or not an example about how to leverage Debezium as a CDC source and to feed a Flink Table (From MySQL for example). Best, Flavio

Re: Questions about user doc.

Hi Vishwas, > I am guessing this means that Flink executes successive tasks from different pipelines successively right ? As the document described, "Note that Flink often executes successive tasks concurrently: For Streaming programs, that happens in any case, but also for batch programs, it

yarn-session vs cluster per job for streaming jobs

Hi, I'm looking for advice on how to run flink streaming jobs on Yarn cluster in production environment. I tried in testing environment both approaches with HA mode, namely yarn session + multiple jobs vs cluster per job, both seems to work for my cases, with slight preference of yarn session

Re: Automatic deployment of new version of streaming stateful job

Hi Marc, thanks a lot for the tool. Unfortunately, I could not direcly use it, but I will take couple of ideas and will implement my own script. Nevertherless, I'm really surprised that such functionality doesn't exist out of the box. Regards, Maxim. On Tue, Jul 16, 2019 at 9:22 AM Marc

Re: [DISCUSS] Create a Flink ecosystem website

Hey all, Daryl and I have great news to share. We are about to finish adding the basic features to the ecosystem page. We are at a stage where it is ready to be reviewed and made public. You can either check out a development instance of the ecosystem page here:

Re: Does Flink support raw generic types in a merged stream?

Hi Chesnay, Yes, but the actual use case needs to support more than 2 streams, so if I go down the Either route then I have arbitrarily sized nested Eithers, i.e. Either, C> etc, which gets pretty messy very quickly. Many thanks, John Sent from my iPhone On 17 Jul 2019, at 13:29, Chesnay

Re: table toRetractStream missing last record and adding extra column (True)

Question 1:- I did tired map function end up having issue ( https://stackoverflow.com/questions/57063249/flink-scala-notinferedr-in-scala-type-mismatch-mapfunctiontuple2boolean-row-i ) I am trying to convert a Tuple[Boolean,Row] to Row using map function, I am getting this error asking me for

Re: table toRetractStream missing last record and adding extra column (True)

Hi Sri, Question1: You can use a map to filter the "true", i.e, ds.map(_._2). Note, it's ok to remove the "true" flag for distinct as it does not generate updates. For other query contains updates, such as a non-window group by, we should not filter the flag or the result is not correct.

Re: org.apache.flink.table.api.TableException: Only tables that originate from Scala DataStreams can be converted to Scala DataStreams.

Hi Sri, For scala jobs, we should import the corresponding scala Environment and DataStream. e.g, import org.apache.flink.streaming.api.scala.{DataStream, StreamExecutionEnvironment} import org.apache.flink.table.api.scala.StreamTableEnvironment See example here[1]. Best, Hequn [1]

Re: Does Flink support raw generic types in a merged stream?

Have you looked at org.apache.flink.types.Either? If you'd wrap all elements in both streams before the union you should be able to join them properly. On 17/07/2019 14:18, John Tipper wrote: Hi All, Can I union/join 2 streams containing generic classes, where each stream has a different

Does Flink support raw generic types in a merged stream?

Hi All, Can I union/join 2 streams containing generic classes, where each stream has a different parameterised type? I'd like to process the combined stream of values as a single raw type, casting to a specific type for detailed processing, based on some information in the type that will

unsubscribe

unsubscribe Thanks & Regards, Kailash Kota Product Development | JDA Software Pvt Ltd. Ph: +91 80 6101 8649 Think BIG > Start small > Run fast! Download JDA's AI/ML buyers guide and get started on your autonomous SCM journey! To the extent permitted by law, we may

[Table API] ClassCastException when converting a table to DataStream

Hello, Consider the following snippet: > Table sourceTable = getKafkaSource0(tEnv); > DataStream stream = tEnv.toAppendStream(sourceTable, Row.class) > > * .map(a -> a) .returns(sourceTable.getSchema().toRowType());* > stream.print(); > where sourceTable.printSchema()