Re:Re: Re: flink on yarn 指定节点开启 yarn session 报错

hi，我确保，端口没占用的情况下更换rest.port: 8072，且更改了配置： #== # HistoryServer #== # The HistoryServer is started and stopped via bin/historyserver.sh

Re: ***UNCHECKED*** Error while confirming Checkpoint

Hi, I want to raise this question again, since I have had this exception on my production job. The exception is as follows > 2019-11-27 14:47:29 java.lang.RuntimeException: Error while confirming checkpoint > at org.apache.flink.runtime.taskmanager.Task$2.run(Task.java:1205) > at

Re: CoGroup SortMerger performance degradation from 1.6.4 - 1.9.1?

Hi, > @yingjie Do you have any idea how much memory will be stolen from OS when > using mmap for data reading? I think this is bounded only by the size of the written data. Also it will not be “stolen from OS”, as kernel is controlling the amount of pages residing currently in the RAM

Re: SQL Performance

Hi Nicholas, You can take a look to JDBCSinkFunction and JDBCUpsertSinkFunction. They can set something like flush batch max size just like vino mentioned. JDBCUpsertOutputFormat also has a async thread to flush batch to JDBC to avoid too high delay. In this way, the throughput can be improved

Re:回复: flink on yarn 的 kerberos 认证问题

HBase认证需要ZooKeeper和Kerberos安全认证,跟ZooKeeper认证“jaas.conf”文件也加载进去了吗? LoginUtil.setJaasConf(ZOOKEEPER_DEFAULT_LOGIN_CONTEXT_NAME, userName, userKeytabFile); LoginUtil.setZookeeperServerPrincipal(ZOOKEEPER_SERVER_PRINCIPAL_KEY, ZOOKEEPER_DEFAULT_SERVER_PRINCIPAL); LoginUtil.login(userName,

回复: flink on yarn 的 kerberos 认证问题

我们kafka 可以正常认证、消费数据，认证hbase 和kafka 好像不一样，我们是不认证 读不到数据，认证了，任务又提交不到 yarn 上去了 如下： >看过对应位置的代码，将 “Hadoop.security.authentication = kerberos” >参数添加到 Hadoop 的 配置文件中（注： 使用 simple 认证的 hadoop集 >群使用 amberi 部署 的 hdp 集群，不开启 Kerberos 认证 参数 “Hadoop.security. >authentication” 的值为 simple ），使程序认证通过，但是

Re: Flink consume Kafka with schema registry

Hi, Can not agree more, if it is supported. Because we need, a On Wed, Nov 27, 2019 at 11:00 AM 朱广彬 wrote: > I have the same problem these days. > > I finally customize avro related serde schema for supporting schema > registry. > > The root cause is that, when serialization , the avro

Re: [DISCUSS] Make Managed Memory always off-heap (Adjustment to FLIP-49)

Hi Stephan, +1 to default have off-heap managed memory. >From the perspective of batch, In our long-term performance test and online practice: - There is no significant difference in performance between heap and off-heap memory. If it is a heap object, the JVM has many opportunities to optimize

Re: CoGroup SortMerger performance degradation from 1.6.4 - 1.9.1?

Hi, I’ve met the exactly same problem recently and solved it in Piotr’s way. @zhijiang, I didn’t see any oom error thrown by JVM (I’m not sure this can be thrown if yarn decides to kill it in a mandatory way). According to our monitoring system, the overusage of memory is from JVM directy

Re: Re: flink on yarn 指定节点开启 yarn session 报错

确实是 historyserver.web.port 的默认值，但我记得 HistoryServer 默认是不启动的 总之可以确认下问题能否稳定复现，失败的瞬间端口有无被占用，以及换个端口能不能起来 如果还不行再找找其他原因 Best, tison. Yangze Guo 于2019年11月27日周三 上午11:23写道： > 8082我记得是historyserver.web.port的默认值，很可能是冲突了改成8081呢？ > > Best, > Yangze Guo > > On Wed, Nov 27, 2019 at 11:13 AM 杨浩程 wrote: > >

Re: Re: flink on yarn 指定节点开启 yarn session 报错

8082我记得是historyserver.web.port的默认值，很可能是冲突了改成8081呢？ Best, Yangze Guo On Wed, Nov 27, 2019 at 11:13 AM 杨浩程 wrote: > > > 好的，更改过的配置如下： > > #==

Re:Re: flink on yarn 指定节点开启 yarn session 报错

好的，更改过的配置如下： #== # Rest & web frontend #== # The port to which

Re:Re: flink on yarn 指定节点开启 yarn session 报错

tison 你好， 我netstat查看了想要指定的节点的端口情况，8082没有被占用。所以有点困惑这个报错。 在 2019-11-27 10:59:52，"tison" 写道： >8082 端口被占用了吧？ > >Best, >tison. > > >杨浩程 于2019年11月27日周三 上午10:56写道： > >> 各位大佬好！ >>请教各位个问题：使用的flink 版本1.9.0。测试flink on yarn 指定节点 开启yarn session 会话。 >>希望yarnsession开启的集群

Re: flink on yarn 指定节点开启 yarn session 报错

您好，目前ML不支持图片，能将配置贴上来么？ 另外看报错信息像是8082端口被占用导致rest服务起不起来 Best, Yangze Guo On Wed, Nov 27, 2019 at 10:56 AM 杨浩程 wrote: > 各位大佬好！ >请教各位个问题：使用的flink 版本1.9.0。测试flink on yarn 指定节点 开启yarn session 会话。 >希望yarnsession开启的集群 jobmanager开启在我指定的节点上。 >更改的配置如下： > 报错信息如下： > >

Re: Flink consume Kafka with schema registry

I have the same problem these days. I finally customize avro related serde schema for supporting schema registry. The root cause is that, when serialization , the avro record with schema registry restriction is different with “original” avro record without schema registry restriction . The

flink on yarn 指定节点开启 yarn session 报错

各位大佬好！ 请教各位个问题：使用的flink 版本1.9.0。测试flink on yarn 指定节点 开启yarn session 会话。 希望yarnsession开启的集群 jobmanager开启在我指定的节点上。 更改的配置如下： 报错信息如下： === 2019-11-27 10:35:45,640 INFO

Flink consume Kafka with schema registry

Hi, I have occur the problem that the data in Kakfa is formatted as avro with schema register server. I found that is not easy to consume this topic easy, the provided kafka does not support this, and I do not want to write a new kafka source, is there any way to using provided kafka source to

Re: stack job on fail over

Hi Nick, Yes, reducing heartbeat timeout is not a perfect solution. It just alleviates the pain a bit. I'm wondering my guess is right or not. Is it caused by heartbeat detection? Does it help with an elegant way of shutting down? Thanks, Biao /'bɪ.aʊ/ On Tue, 26 Nov 2019 at 20:22, Nick

Re: How to custom (or use) a window to specify everyday's beginning as watermark?

Hi Rock, There is an inaccurate description in last response. I don't think a watermark of 0 clock is needed to get the accurate calculation result. The watermark of 0 clock only helps to generate the result you want immediately. Thanks, Biao /'bɪ.aʊ/ On Tue, 26 Nov 2019 at 18:10, Biao Liu

Re: SQL Performance

Hi Nick, Can you provide more details? Are you using JDBCOutputFormat? If yes, can `JDBCOutputFormatBuilder#setBatchInterval` help you? Best, Vino Nicholas Walton 于2019年11月26日周二 下午9:20写道： > I’m streaming records down to an Embedded Derby database, at a rate of > around 200 records per second.

[DISCUSS] Make Managed Memory always off-heap (Adjustment to FLIP-49)

Hi all! Yesterday, some of the people involved in FLIP-49 had a long discussion about managed memory in Flink. Particularly, the fact that we have managed memory either on heap or off heap and that FLIP-49 introduced having both of these types of memory at the same time. ==> What we want to

Re: 退出

退出 jc-zho...@chinaunicom.cn 发件人： 杨荣 发送时间： 2019-11-27 09:14 收件人： user-zh@flink.apache.org 主题： 退出 退出 如果您错误接收了该邮件，请通过电子邮件立即通知我们。请回复邮件到 hqs-s...@chinaunicom.cn，即可以退订此邮件。我们将立即将您的信息从我们的发送目录中删除。 If you

退出

退出

What happens to a Source's Operator State if it stops being initialized and snapshotted? Accidentally exponential?

Hi, Some context: after a refactoring, we were unable to start our jobs. They started fine and checkpointed fine, but once the job restarted owing to a transient failure, the application was unable to start. The Job Manager was OOM'ing (even when I gave them 256GB of ram!). The `_metadata` file

Re: Per Operator State Monitoring

Hi Aaron, I don't think we have such fine grained metrics on per operation state size, but from your description that "YARN kills containers who are exceeding their memory limits", I think the root cause is not the state size but related to the memory consumption of the state backend. My guess

Re: SQL for Avro GenericRecords on Parquet

Hi Hanan, After investigating the issue by using the test case you provided, I think there is a big in it. Currently, the parquet predicts push down use the predicate literal type to construct the FilterPredicate. The issue happens when the data type of value in predicate inferred from SQL

Re: CoGroup SortMerger performance degradation from 1.6.4 - 1.9.1?

Thanks for the confirmation, I’ve created Jira ticket to track this issue [1] https://issues.apache.org/jira/browse/FLINK-14952 Piotrek > On 26 Nov 2019, at 11:10, yingjie wrote: > > The new BlockingSubpartition implementation in 1.9 uses

SQL Performance

I’m streaming records down to an Embedded Derby database, at a rate of around 200 records per second. I’m certain Derby can sustain a higher throughput than that, if I could buffer the records but it seems that I’m writing each record as soon as it arrives and as a single transaction which is

回复：flink检查点状态大小

你这个用了distinct的话，肯定就是针对全局的id进行distinct了，设置state也无效。 建议自己使用bigmap或者hyperlog算法实现一个distinct，这样可以节省内存 原始邮件 发件人:谷歌-akulakuzhan...@akulaku.com 收件人:user-zh@flink.apache.orguser...@flink.apache.org 发送时间:2019年11月26日(周二) 20:05 主题:flink检查点状态大小

flink检查点状态大小

streamTableEnv.getConfig().setIdleStateRetentionTime(Time.minutes(15), Time.minutes(20)); 我在程序中设置状态保留时间，然后用全局group进行计算，但是过期状态没有清理导致状态也来越大，最终内存溢出，请问这是什么原因导致的 运行SQL select count(distinct id) as user_count,adjust_time from (select data.f13 as country_id,data.f1 as

Re: Pre-process data before it hits the Source

ok. I am sorry, I thought that was you that said this. Maybe it is just a matter of expression that made the question confused. But, yes. In the source function something can be done. Not before. *--* *-- Felipe Gutierrez* *-- skype: felipe.o.gutierrez* *--*

Re: Pre-process data before it hits the Source

Hi Felipe, >> But you said, "before it hits the Source". I did not say this. Vijay said it. About this question, he may not think about customizing the source connector. If he does not try to find a solution in the Flink domain. Why he asked Flink questions and pasted Flink program? IMO, It's

ArrayIndexOutOfBoundException on checkpoint creation

Hi, We have a pipeline with a custom ProcessFunction and state (see [1], implemented as suggested by Fabian with a ValueState and ValueState>) The behavior of that function works fine in our unittests and with low load in our test environment (100.000 records per minute). On the production

Re: How to custom (or use) a window to specify everyday's beginning as watermark?

Hi Rock, >From my understanding, what you want is a one-day time based window which start at 0 clock. Actually the one-day time-based window (like Jack mentioned) starts at the beginning of day (0:00). You don't need to do anything special. If you are using event time window (since you mentioned

Re: CoGroup SortMerger performance degradation from 1.6.4 - 1.9.1?

The new BlockingSubpartition implementation in 1.9 uses mmap for data reading by default which means it steals memory from OS. The mmaped region memory is managed by JVM, so there should be no OutOfMemory problem reported by JVM and the OS memory is also not exhausted, so there should be no kernal

Re: How to custom (or use) a window to specify everyday's beginning as watermark?

Hi Rock, Sorry, I don't fully understand what you want. If you want a tumbling window which covers one day, you can use `KeyedStream#timeWindow(Time.days(1))` which covers from UTC 00:00~24:00. Best, Jark On Tue, 26 Nov 2019 at 17:20, Caizhi Weng wrote: > Hi Rock, > > I think you can write

Kafka Offset commit failed on partition

Hello, Since the last update to the universal Kafka connector, I'm getting the following error fairly often. /2019-11-18 15:42:52,689 ERROR org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-10, groupId=srx-consumer-group] Offset commit failed on

Re: stack job on fail over

Hi Nick, I guess the reason is your Flink job manager doesn't detect the task manager is lost until heartbeat timeout. You could check the job manager log to verify that. Maybe a more elegant way of shutting down task manager helps, like through "taskmanager.sh stop" or "kill" command without 9

Re: How to custom (or use) a window to specify everyday's beginning as watermark?

Hi Rock, I think you can write your own trigger which fires when the date of the process time of the current record is different from that of the last record. Pinging @Jark Wu for a more professional answer. Rock 于2019年11月26日周二 下午3:37写道： > I need my job to aggregator every device's mertic as

Re: Pre-process data before it hits the Source

Hi Vino, yes, in the source function it is possible. But you said, "before it hits the Source". So, IMO I think it is outside of the flink workflow. Best, Felipe *--* *-- Felipe Gutierrez* *-- skype: felipe.o.gutierrez* *--* *https://felipeogutierrez.blogspot.com

Re: Problem loading JDBC driver

Hi Nick, The "Test" after "org.apache.derby" % "derby" % "10.15.1.3" seems suspicious. Is that intended? Nicholas Walton 于2019年11月26日周二 下午4:46写道： > Hi, > > *I have a pipeline which is sinking into an Apache Derby database, but I’m > constantly receiving the error* > >

Re: Pre-process data before it hits the Source

Hi Felipe, Why do you think it's not possible. My thought is we can do the data pre-procession in the source function. If so, source function would contain consume upstream events then do pre-processing then emits to the downstream. Best, Vino Felipe Gutierrez 于2019年11月26日周二 下午4:56写道： > I

Re: Question about to modify operator state on Flink1.7 with State Processor API

Hi Vino/Seth, Thanks Vino and Seth, changing the UID and setting offset manually is a solution, but the pin point is we have tons of applications(owned by other users) running on our platform, so it will be inefficient to do it manually, and the most difficult part is to let users to change their

Re: Pre-process data before it hits the Source

I am afraid that this is not possible in FLink, since the entry point of all transformation is the source function. Everything that we can pre-process is in the source function or on the downstream operators. If you want to pre-process something before the data hits the source you will have to

Problem loading JDBC driver

Hi, I have a pipeline which is sinking into an Apache Derby database, but I’m constantly receiving the error java.lang.IllegalArgumentException: JDBC driver class not found. The Scala libraries I’m loading are val flinkDependencies = Seq( "org.apache.flink" %% "flink-scala" % flinkVersion ,

flink1.9.1 rest_api cancel作业问题

根据文档 | /jobs/:jobid | | Verb: PATCH | Response code: 202 Accepted | | Terminates a job. | 但是发起patch请求后, 返回: HTTP/1.1 501 Method PATCH is not defined in RFC 2068 and is not supported by the Servlet API 任务是 yarn-session 模式

stack job on fail over

Hi i have a standalone cluster with 3 nodes and rocksdb backend when one task manager fails ( the process is being killed) it takes very long time until the job is totally canceled and a new job is resubmitted i see that all slots on all nodes are being canceled except from the slots of the dead