Re:Flink Web UI display nothing in k8s when use ingress

Hi lake:    Ok, Show the jobmanager pod logs, Can you see the jm pods is running ok? Try use cube-proxy, or NodePort, That you can see the webUI? Best,Ouywl On

Re: Flink Web UI display nothing in k8s when use ingress

In my thought , I think I should config the correct flink jobserver for flink task LakeShen 于2020年3月4日周三 下午2:07写道： > Hi community, > now we plan to move all flink tasks to k8s cluster. For one flink > task , we want to see this flink task web ui . First , we create the k8s > Service to

Re: Flink Web UI display nothing in k8s when use ingress

In my thought , I think I should config the correct flink jobserver for flink task LakeShen 于2020年3月4日周三 下午2:07写道： > Hi community, > now we plan to move all flink tasks to k8s cluster. For one flink > task , we want to see this flink task web ui . First , we create the k8s > Service to

Flink Web UI display nothing in k8s when use ingress

Hi community, now we plan to move all flink tasks to k8s cluster. For one flink task , we want to see this flink task web ui . First , we create the k8s Service to expose 8081 port of jobmanager, then we use ingress controller so that we can see it outside.But the flink web like this :

Flink Web UI display nothing in k8s when use ingress

Hi community, now we plan to move all flink tasks to k8s cluster. For one flink task , we want to see this flink task web ui . First , we create the k8s Service to expose 8081 port of jobmanager, then we use ingress controller so that we can see it outside.But the flink web like this :

?????? ????Flink1.10.0????hive??source??????????

??sql??sql??hive??sql?? ??2020??03??4??

Re: 使用Flink1.10.0读取hive时source并行度问题

Hi jun, Jira: https://issues.apache.org/jira/browse/FLINK-16413 FYI Best, Jingsong Lee -- From:JingsongLee Send Time:2020年3月3日(星期二) 19:06 To:Jun Zhang <825875...@qq.com>; user-zh@flink.apache.org Cc:user-zh@flink.apache.org ;

Re: JobMaster does not register with ResourceManager in high availability setup

Hi Abhinav, The JobMaster log "Connecting to ResourceManager ..." is printed after JobMaster retrieve ResourceManager address from ZooKeeper. In your case, I assume there's some ZK problem that JM cannot resolve RM address. Have you confirmed whether the ZK pods are recovered after the second

Re: Should I use a Sink or Connector? Or Both?

John is right. Could you provide more detailed code? So that we can help to investigate. Best, Jark On Wed, 4 Mar 2020 at 06:20, John Smith wrote: > The sink if for Streaming API, it looks like you are using SQL and tables. > So you can use the connector to output the table result to Elastic.

Re: CliFrontend 未优先加载用户jar包中的class

也是一种 hack 的方法，不过社区肯定不能在 master 上这么搞就是了（x Best, tison. aven.wu 于2020年3月3日周二 下午4:44写道： > 感谢回答 > 后来我查了Flink run脚本的classpath设置，我修改了脚本将我的jar包指定在flink classpath的最前面得以解决问题 > > Best > Aven > > 发件人: tison > 发送时间: 2020年3月3日 14:16 > 收件人: user-zh > 主题: Re: CliFrontend 未优先加载用户jar包中的class > > >

Very large _metadata file

Per the documentation: "The meta data file of a Savepoint contains (primarily) pointers to all files on stable storage that are part of the Savepoint, in form of absolute paths." I somehow have a _metadata file that's 1.9GB. Running *strings *on it I find 962 strings, most of which look like

Re: Should I use a Sink or Connector? Or Both?

The sink if for Streaming API, it looks like you are using SQL and tables. So you can use the connector to output the table result to Elastic. Unless you want to convert from table to stream first. On Tue, 3 Mar 2020 at 16:25, Castro, Fernando C. wrote: > Hello folks! I’m new to Flink and data

Re: StreamingFileSink Not Flushing All Data

Hi all, Thanks for the docs pointer/ FLIP Rafi, and the workaround strategy Kostas -- strange though, as I wasn't using a bounded source when I first ran into this issue. I have updated the example repo to use an unbounded source[1], and the same file corruption problems remain. Anything else I

JobMaster does not register with ResourceManager in high availability setup

Hi, We recently came across an issue where JobMaster does not register with ResourceManager in Fink high availability setup. Let me share the details below. Setup * Flink 1.7.1 * K8s * High availability mode with a single Jobmanager and 3 zookeeper nodes in quorum. Scenario *

Should I use a Sink or Connector? Or Both?

Hello folks! I’m new to Flink and data streaming in general, just initial FYI ;) I’m currently doing this successfully: 1 - streaming data from Kafka in Flink 2 - aggregating the data with Flink’s sqlQuery API 3 - outputting the result of #2 into STDOUT via toRetreatStream() My objective is to

How to print the aggregated state everytime it is updated?

Hi All, I have a custom aggregated state that is represent by Set and I have a stream of values coming in from Kafka where I inspect, compute the custom aggregation and store it in Set. Now, I am trying to figureout how do I print the updated value everytime this state is updated? Imagine I have

Re: Flink Session Windows State TTL

Added flink_oom_exception.txt as originally forgot to attach it -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Flink Session Windows State TTL

Hi all,We have an Apache Flink application which generates player sessions based on player events keyed by playerId. Sessions are based on EventTime. A session is created on first event event for that player and closes if there are 30 mins of inactivity. Events are merged in our custom

Re: Single stream, two sinks

If I understand correctly he wants the HTTP result in the DB. So I do not think side output works here. The DB would have to be the sink. Also sinks in Flink are the final destination. So it would have to be RabbitMQ -> Some Cool Business Logic Operators Here > Async I/O HTTP Operator

Re: java.util.concurrent.ExecutionException

Hi, Thanks for getting back, and I am glad that you were able to resolve the issue. The root cause in the stacktrace you posted also indicates a problem related to Kafka: Caused by: org.apache.flink.kafka.shaded.org.apache.kafka.common.errors.TimeoutException: Timeout of 6ms expired

RE: Building with Hadoop 3

Hello, Have you shared it somewhere on the web already? Best, Arnaud De : vino yang Envoyé : mercredi 4 décembre 2019 11:55 À : Márton Balassi Cc : Chesnay Schepler ; Foster, Craig ; user@flink.apache.org; d...@flink.apache.org Objet : Re: Building with Hadoop 3 Hi Marton, Thanks for your

Re: java.util.concurrent.ExecutionException

Hi Gary, This has to do with my Kafka. After restarting Kafka it seems to work fine! Thanks! On Tue, Mar 3, 2020 at 8:18 AM kant kodali wrote: > The program finished with the following exception: > > > org.apache.flink.client.program.ProgramInvocationException: The main > method caused an

Re: Flink Stream Sink ParquetWriter Failing with ClassCastException

Hi Anuj, if you use the exact same schema with which the data has been written for reading and if there is no bug in the parquet Avro support, then it should indeed not fail. Hence, I suspect that the producer of your data might produce slightly different Avro records compared to what Parquet is

Re: java.util.concurrent.ExecutionException

The program finished with the following exception: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: org.apache.flink.client.program.ProgramInvocationException: Job failed (JobID: f57b682f5867a8bf6ff6e1ddce93a1ab) at

Re: java.util.concurrent.ExecutionException

Hi, Can you post the complete stacktrace? Best, Gary On Tue, Mar 3, 2020 at 1:08 PM kant kodali wrote: > Hi All, > > I am just trying to read edges which has the following format in Kafka > > 1,2 > 1,3 > 1,5 > > using the Table API and then converting to DataStream of Edge Objects and >

Re: Providing hdfs name node IP for streaming file sink

Thanks Yang. Going with setting the HADOOP_CONF_DIR in the flink application. It integrates neatly with flink. Best, Nick. On Mon, Mar 2, 2020 at 7:42 PM Yang Wang wrote: > It may work. However, you need to set your own retry policy(similar as > `ConfiguredFailoverProxyProvider` in hadoop). >

[Survey] Default size for the new JVM Metaspace limit in 1.10

Hi All, Recently, FLIP-49 [1] introduced the new JVM Metaspace limit in the 1.10 release [2]. Flink scripts, which start the task manager JVM process, set this limit by adding the corresponding JVM argument. This has been done to properly plan resources. especially to derive container size for

Re: zookeeper.connect is not needed but Flink requires it

Hi Kant, You are right. It is not needed since Kafka 0.9+. We already have an issue to make it optional. https://issues.apache.org/jira/browse/FLINK-16125 Best, Jark On Tue, 3 Mar 2020 at 20:17, kant kodali wrote: > Hi All, > > The zookeeper.connect is not needed for KafkaConsumer or

Re: Operator latency metric not working in 1.9.1

Thanks, that makes sense! In addition, I've just found the reason for this in the code: This is 1.5 (default value is 2000L): https://github.com/apache/flink/blob/release-1.5/flink-core/src/main/java/org/apache/flink/configuration/MetricOptions.java#L109 This is 1.9 (default value is 0L)

Re: Alink and Flink ML

Hi Flavio, I am looping in Becket (cc'ed) who might be able to answer your question. Best, Gary On Tue, Mar 3, 2020 at 12:19 PM Flavio Pompermaier wrote: > Hi to all, > since Alink has been open sourced, is there any good reason to keep both > Flink ML and Alink? > From what I understood

Re: Unable to recover from savepoint and checkpoint

Hi Puneet, Can you describe how you validated that the state is not restored properly? Specifically, how did you introduce faults to the cluster? Best, Gary On Tue, Mar 3, 2020 at 11:08 AM Puneet Kinra < puneet.ki...@customercentria.com> wrote: > Sorry for the missed information > > On

????Java????????????????

hi all??Java?? https://s2.ax1x.com/2020/03/03/34yyvT.png https://s2.ax1x.com/2020/03/03/34y5P1.png ??

Re: Operator latency metric not working in 1.9.1

Hi, There is a release note for Flink 1.7 that could be relevant for you [1] Granularity of latency metrics The default granularity for latency metrics has been modified. To restore the previous behavior users have to explicitly set the granularity to subtask. Best, Gary [1]

zookeeper.connect is not needed but Flink requires it

Hi All, The zookeeper.connect is not needed for KafkaConsumer or KafkaAdminClient however Flink requires it. You can also see in the Flink TaskManager logs the KafkaConsumer is not recognizing this property anyways. bsTableEnv.connect( new Kafka() .property("bootstrap.servers",

Alink and Flink ML

Hi to all, since Alink has been open sourced, is there any good reason to keep both Flink ML and Alink? >From what I understood Alink already contains the best ML implementation available for Flink..am I wrong? Maybe it could make sense to replace the current Flink ML with that of Alink..or is

Re: 使用Flink1.10.0读取hive时source并行度问题

Hi jun, 很好的建议~ 这是一个优化点~ 可以建一个JIRA Best, Jingsong Lee -- From:Jun Zhang <825875...@qq.com> Send Time:2020年3月3日(星期二) 18:45 To:user-zh@flink.apache.org ; JingsongLee Cc:user-zh@flink.apache.org ; like Subject:回复：

Re: Hive Source With Kerberos认证问题

datanucleus是在HMS端使用的，如果没有datanucleus会报错的话说明你的代码在尝试创建embedded metastore。这是预期的行为么？我理解你们应该是有一个远端的HMS，然后希望HiveCatalog去连接这个HMS吧？ On Tue, Mar 3, 2020 at 4:00 PM 叶贤勋 wrote: > hive conf应该是对的，前面UserGroupInfomation登录时都是成功的。 > datanucleus的依赖不加的话，会报claas not found等异常。 > 1、java.lang.ClassNotFoundException:

?????? ????Flink1.10.0????hive??source??????????

hi??jinsong?? ??10?? ??sql select * from mytable limit 1; hive??mytable??10??10?? ??2020??03??2?? 16:38??JingsongLee

Unable to recover from savepoint and checkpoint

Hi Stuck with the simple program regarding the checkpointing Flink version I am using 1.10.0 *Here I have created DummySource for testing* *DummySource* package com.nudge.stateful; import org.apache.flink.api.java.tuple.Tuple2; import

回复： Hive Source With Kerberos认证问题

这是我和flink社区沟通的记录，你可以看下。 | | 叶贤勋 | | yxx_c...@163.com | 签名由网易邮箱大师定制 在2020年03月3日 16:00，叶贤勋 写道： hive conf应该是对的，前面UserGroupInfomation登录时都是成功的。 datanucleus的依赖不加的话，会报claas not found等异常。 1、java.lang.ClassNotFoundException: org.datanucleus.api.jdo.JDOPersistenceManagerFactory 2、Caused by:

Re: Operator latency metric not working in 1.9.1

Thanks for the response. In 1.5 the docs also state that it should be enabled [1], however, it always worked without setting latencyTrackingInterval [1] https://ci.apache.org/projects/flink/flink-docs-release-1.5/monitoring/metrics.html#latency-tracking -- Sent from:

回复: CliFrontend 未优先加载用户jar包中的class

感谢回答 后来我查了Flink run脚本的classpath设置，我修改了脚本将我的jar包指定在flink classpath的最前面得以解决问题 Best Aven 发件人: tison 发送时间: 2020年3月3日 14:16 收件人: user-zh 主题: Re: CliFrontend 未优先加载用户jar包中的class https://github.com/apache/flink/commit/0f30c263eebd2fc3ecbeae69a4ce9477e1d5d774 Best, tison. tison 于2020年3月3日周二

Re: Flink EventTime Processing Watermark is always coming as 9223372036854725808

Thanks, Robert for mentioning this, I will take care of it in future posts. I am able to figure out the issue. When I disable checkpoint then the watermark is getting updated and its working. I need to understand 2 things : 1. Please help to understand what is happening when I enable

回复： Hive Source With Kerberos认证问题

hive conf应该是对的，前面UserGroupInfomation登录时都是成功的。 datanucleus的依赖不加的话，会报claas not found等异常。 1、java.lang.ClassNotFoundException: org.datanucleus.api.jdo.JDOPersistenceManagerFactory 2、Caused by: org.datanucleus.exceptions.NucleusUserException: There is no available StoreManager of type "rdbms". Please