Re: [ANNOUNCE] Apache Flink 1.18.0 released

2023-10-26 文章 Yangze Guo
Great work! Congratulations to everyone involved! Best, Yangze Guo On Fri, Oct 27, 2023 at 10:23 AM Qingsheng Ren wrote: > > Congratulations and big THANK YOU to everyone helping with this release! > > Best, > Qingsheng > > On Fri, Oct 27, 2023 at 10:18 AM Benchao Li wr

Re: Flink资源动态分配

2021-11-14 文章 Yangze Guo
可以尝试下Reactive mode.[1] 扩缩容操作仍需要外部干预。 [1] https://nightlies.apache.org/flink/flink-docs-master/zh/docs/deployment/elastic_scaling/ Best, Yangze Guo On Mon, Nov 15, 2021 at 3:41 PM 疾鹰击皓月 <1764232...@qq.com.invalid> wrote: > > 您好: > > Flink工程白天和晚上数据流量差距非常巨大,如果并行度设置的低无法应对白天的流量

Re: flink 1.13.1 通过yarn-application运行批应用,处理mysql源一亿条数据到hive,发现需要配置16G+的Taskmangaer内存

2021-11-04 文章 Yangze Guo
失败的原因呢?有没有报错栈和日志? Best, Yangze Guo On Thu, Nov 4, 2021 at 4:01 PM Asahi Lee <978466...@qq.com.invalid> wrote: > > hi! > 我通过flink sql,将mysql的一亿条数据传输到hive库中,通过yarn-application方式运行,结果配置16G的内存,执行失败!

Re: flink on native kubernetes作业cpu实际使用量与请求量相差太大问题

2021-09-16 文章 Yangze Guo
可能是cpu过少导致tm启动慢,超过其注册限定时间,可以尝试调大resourcemanager.taskmanager-registration.timeout Best, Yangze Guo On Fri, Sep 17, 2021 at 9:36 AM casel.chen wrote: > > 我们使用Flink运行实时作业在Kubernetes,发现作业实际使用的CPU资源远远小于作业请求量,但是将作业请求量降低后发现作业启动不了。请问这是个案还是正常情况? > 例如,我们一个作业请求了0.5个cpu,但实际使用量只有0.09左右,修改请求为0.2个c

Re: taskmanager数问题请教

2021-08-05 文章 Yangze Guo
现在yarn上都是按需申请的 Best, Yangze Guo On Fri, Aug 6, 2021 at 10:31 AM 上官 wrote: > > 1.13版本中yarn模式提交时 -yn好像不能用了,请问现在要如何指定容器(taskmanager)数量?

Re: 问题咨询

2021-05-09 文章 Yangze Guo
Hi, 看日志application已经提交到yarn了,但是am没有调度起来,看一下Yarn界面application_1620481460888_0003 这个application的状态是什么?有没有报错 Best, Yangze Guo On Mon, May 10, 2021 at 9:41 AM wyinj...@126.com wrote: > > 您好,在使用flink on YARN的时候遇到了问题,寻求帮助,YARN集群搭建完成之后,运行flink命令如下: > ./bin/flink run -t yarn-per-job -

Re: flink1.12.1 Sink数据到ES7,遇到 Invalid lambda deserialization 问题

2021-04-15 文章 Yangze Guo
可以参考下[1], 如果是相同的问题,将依赖改为flink-connector-elasticsearch [1] https://issues.apache.org/jira/browse/FLINK-18857 Best, Yangze Guo On Fri, Apr 16, 2021 at 10:43 AM Yangze Guo wrote: > > 有完整报错栈或者日志能发下么? > > Best, > Yangze Guo > > On Fri, Apr 16, 2021 at 9:33 AM william <

Re: flink1.12.1 Sink数据到ES7,遇到 Invalid lambda deserialization 问题

2021-04-15 文章 Yangze Guo
有完整报错栈或者日志能发下么? Best, Yangze Guo On Fri, Apr 16, 2021 at 9:33 AM william <712677...@qq.com> wrote: > > > > > > -- > Sent from: http://apache-flink.147419.n8.nabble.com/

Re: Does it support gpu coding in flink?

2021-04-12 文章 Yangze Guo
/src/main/java/com/alibaba/flink/ml/examples/tensorflow/mnist Best, Yangze Guo On Mon, Apr 12, 2021 at 2:35 PM 张颖 wrote: > > HI,I am running a tf inference task on my cluster,but I flind it took so long > a time to get response, becase it is a bert model and I run it on cpu >

Re: [ANNOUNCE] Apache Flink 1.12.1 released

2021-01-19 文章 Yangze Guo
Thanks Xintong for the great work! Best, Yangze Guo On Tue, Jan 19, 2021 at 4:47 PM Till Rohrmann wrote: > > Thanks a lot for driving this release Xintong. This was indeed a release with > some obstacles to overcome and you did it very well! > > Cheers, > Till > > On T

Re: yarn Per-Job Cluster Mode提交任务时 通过cli指定的内存参数无效

2021-01-17 文章 Yangze Guo
请问这个路径是你本地的路径么?需要client端能根据这个路径找到jar包 Best, Yangze Guo On Mon, Jan 18, 2021 at 10:34 AM 刘海 wrote: > > 你好 > 根据你的建议我试了一下 > 将提交命令改为: ./bin/flink run -d -t yarn-per-job -tm 1536 -jm 3072 -D > jobmanager.memory.process.size=1.5GB -D taskmanager.memory.process.size=3GB > -D

Re: yarn Per-Job Cluster Mode提交任务时 通过cli指定的内存参数无效

2021-01-17 文章 Yangze Guo
Hi, 请使用 -D -tm -jm 不需要加y前缀 Best, Yangze Guo Best, Yangze Guo On Mon, Jan 18, 2021 at 9:19 AM 刘海 wrote: > > > 刘海 > liuha...@163.com > 签名由 网易邮箱大师 定制 > 在2021年1月18日 09:15,刘海 写道: > > Hi Dear All, >请教各位一个问题,下面是我的集群配置: > 1、我现在使用的是flink1.12版本; > 2、基于CDH6.3.2搭建的

Re: task manager内存使用问题

2020-12-17 文章 Yangze Guo
1. 加jvm参数可以使用env.java.opts.taskmanager配置 2. 目前tm中没有对heap memory进行slot间细粒度管理,session模式下不支持这种功能 Best, Yangze Guo On Fri, Dec 18, 2020 at 9:22 AM guoliubi...@foxmail.com wrote: > > Hi, > 现在使用的是flink1.12,使用standalone cluster模式运行。 > 在上面运行一个Job内存消耗大,会用满heap然后把整个task manager带崩掉。 > 想问下怎

Re: flink sql 1.12 写数据到elasticsearch,部署问题

2020-12-15 文章 Yangze Guo
需要放 flink-sql-connector-elasticsearch7_2.11-1.12.0.jar Best, Yangze Guo On Wed, Dec 16, 2020 at 11:34 AM cljb...@163.com wrote: > > hi, > flink sql 1.12版本,写数据到elasticsearch时,本地执行正常,部署到服务器上,报如下错误。 > 检查了打的jar包,里面是包含相应的类的,在flink > lib下也已经放了flink-connector-elasticsearch7_2.

Re: flink1.11 datastream elasticsearch sink 写入es需要账号密码验证,目前支持这种操作吗?

2020-12-15 文章 Yangze Guo
1.11版本中尚不支持username和password的设置,这两个配置在1.12中加入了新的es connector[1] [1] https://issues.apache.org/jira/browse/FLINK-18361 Best, Yangze Guo On Wed, Dec 16, 2020 at 11:34 AM 李世钰 wrote: > > flink1.11 datastream elasticsearch sink 写入es需要账号密码验证,目前支持这种操作吗? > elastic

Re: 使用RedisSink无法将读取的Kafka数据写入Redis中

2020-12-06 文章 Yangze Guo
大概率是网络不通,可以检查一下白名单设置 Best, Yangze Guo On Mon, Dec 7, 2020 at 10:28 AM Jark Wu wrote: > > 这个估计和网络和部署有关,建议咨询下华为云的技术支持。 > > On Sun, 6 Dec 2020 at 20:40, 赵一旦 wrote: > > > 连接不上,你的华为云确认和redis服务器连通吗? > > > > 追梦的废柴 于2020年12月6日周日 下午8:35写道: > > > > > 各位

Re: Re: flink tm cpu cores设置

2020-11-10 文章 Yangze Guo
你这个少了一个"v", 应该是yarn.containers.vcores Best, Yangze Guo On Tue, Nov 10, 2020 at 3:43 PM zjfpla...@hotmail.com wrote: > > JM logs里面有 > Loading configuration property: yarn.containers.cores,4 > > > > zjfpla...@hotmail.com > > 发件人: zjfpla...@hotmail.com > 发送

Re: Re: flink tm cpu cores设置

2020-11-08 文章 Yangze Guo
如何确认没有用的呢?能分享一下jm日志么? 另外这个参数实际是否生效也取决于yarn的调度器是否开启了cpu调度 Best, Yangze Guo On Thu, Nov 5, 2020 at 1:50 PM zjfpla...@hotmail.com wrote: > > 这个再flink-conf.yaml中设置过没用 > > > > zjfpla...@hotmail.com > > 发件人: JasonLee > 发送时间: 2020-11-05 13:49 > 收件人: user-zh > 主题:

Re: flink-1.11.2提交到yarn一直处于CREATED中

2020-11-03 文章 Yangze Guo
有更完整的am日志么?需要看一下rm那边资源申请情况。 Best, Yangze Guo On Wed, Nov 4, 2020 at 11:45 AM 酷酷的浑蛋 wrote: > > > > 下面是报错,说是没有资源,但资源是充足的,之后我把版本改为1.11.1,任务就可以运行了 > org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: > Could not allocate the required slot within sl

Re: yarn部署模式kerberos问题

2020-11-03 文章 Yangze Guo
apache.org/projects/flink/flink-docs-master/ops/security-kerberos.html Best, Yangze Guo On Tue, Nov 3, 2020 at 4:17 PM amen...@163.com wrote: > > hi everyone, > > 最近使用flink-1.11.1在通过per-job方式提交任务到yarn队列的时候,碰到了kerberos权限认证问题。 > > 具体描述:在客户端通过Kerberos权限认证指定用户,提交flink任务到yarn队列,正常提交,但是

Re: flink1.11 elasticsearch connector

2020-10-29 文章 Yangze Guo
1.11版本中尚不支持username和password的设置,这两个配置在1.12中加入了新的es connector[1] [1] https://issues.apache.org/jira/browse/FLINK-18361 Best, Yangze Guo On Thu, Oct 29, 2020 at 3:37 PM 赵帅 wrote: > > elasticsearch7.6有账号认证,目前flink1.11 elasticsearch connector sql api如何加入账号认证?

Re: No pooled slot available and request to ResourceManager for new slot failed

2020-10-28 文章 Yangze Guo
Hi, 你job的并发是多少?一共请求了多少个slot? 方便的话最好发一下jm的日志来帮助排查 Best, Yangze Guo On Thu, Oct 29, 2020 at 10:07 AM marble.zh...@coinflex.com.INVALID wrote: > > 大家好。 > > 只有一个job,设置了jm/tm各总内存为3G,一个taskmanager,总共10个slot,为什么还是报这个错

Re: pyflink1.11.0 如果elasticsearch host有访问权限,connector如何写入用户名密码

2020-10-22 文章 Yangze Guo
1.11版本中尚不支持username和password的设置,这两个配置在1.12中加入了新的es connector[1] [1] https://issues.apache.org/jira/browse/FLINK-18361 Best, Yangze Guo On Thu, Oct 22, 2020 at 3:47 PM whh_960101 wrote: > > Hi,各位大佬们:如果要sink数据到elasticsearch host有访问权限,elasticsearch > connector如何写入用户名密码我按照官网里的样例格式来写的,没有找

Re: flink slot资源隔离

2020-10-20 文章 Yangze Guo
Managed Memory是隔离的,Heap,Network都是TM级别共享 Best, Yangze Guo On Wed, Oct 21, 2020 at 10:06 AM 赵一旦 wrote: > > flink slot的资源隔离,内存会真实隔离嘛?cpu肯定不会。

Re: Re: [ANNOUNCE] New PMC member: Dian Fu

2020-08-27 文章 Yangze Guo
Congrats Dian! Best, Yangze Guo On Thu, Aug 27, 2020 at 6:26 PM Zhu Zhu wrote: > > Congratulations Dian! > > Thanks, > Zhu > > Zhijiang 于2020年8月27日周四 下午6:04写道: > > > Congrats, Dian! > > > > --

Re: [ANNOUNCE] Apache Flink 1.10.2 released

2020-08-25 文章 Yangze Guo
Thanks a lot for being the release manager Zhu Zhu! Congrats to all others who have contributed to the release! Best, Yangze Guo On Tue, Aug 25, 2020 at 2:42 PM Dian Fu wrote: > > Thanks ZhuZhu for managing this release and everyone else who contributed to > this release! > >

Re: flink-1.10.1 想用 DDL 入 ES5.6

2020-08-17 文章 Yangze Guo
社区已经在考虑废弃es5的支持了 Best, Yangze Guo On Mon, Aug 17, 2020 at 5:38 PM kcz <573693...@qq.com> wrote: > > 有一个小问题比较好奇,当时社区为什么没有提供ES5的sql功能,是处于什么考虑吗?因为已经提供了ES5 sinK的connect。 > --原始邮件-

Re: Flink参数配置设置不生效

2020-08-16 文章 Yangze Guo
Hi, 请问你的启动命令是什么? 如果指定了 -d ,那会覆盖这个选项 Best, Yangze Guo On Sat, Aug 15, 2020 at 10:52 PM 魏烽 wrote: > > 各位大佬: > > 这个问题没有人遇到过嘛? > > 原始邮件 > 发件人: weifeng > 收件人: user-zh > 发送时间: 2020年8月15日(周六) 09:44 > 主题: Flink参数配置设置不生效 > > > 各位大佬好: > > 在flink-conf.yam

Re: Flink 1.10 on Yarn

2020-08-06 文章 Yangze Guo
日志没有贴成功,taskmanager.memory.jvm-metaspace.size目前是默认配置么? Best, Yangze Guo On Fri, Aug 7, 2020 at 11:38 AM xuhaiLong wrote: > > > > Hi > > > 场景:1 tm 三个slot,run了三个job > > > 三个job 运行的时候 出现了 ck 过程中空指针异常,导致任务一致重启。最终导致`Metaspace` 空间占满,出现 > `java.lang.OutOfMemoryEr

Re: flink-sql-connector-elasticsearch6_2.11_1.10.0 与 flink-connector-elasticsearch6_2.11_1.10.0 并存问题

2020-08-06 文章 Yangze Guo
Hi, flink-sql-connector-elasticsearch6_2.11已经打进了es的相关依赖,直接非provided引入应该是没有问题的,您说的报一样的错是错误栈完全相同么?在ide里面直接跑还是提交任务? Best, Yangze Guo On Fri, Aug 7, 2020 at 9:32 AM JasonLee <17610775...@163.com> wrote: > > Hi > > > 只引入-sql那个包就行了 代码也应该是可以直接用这个包的 > > > | | > JasonLee

Re: [ANNOUNCE] Apache Flink 1.11.1 released

2020-07-22 文章 Yangze Guo
Congrats! Thanks Dian Fu for being release manager, and everyone involved! Best, Yangze Guo On Wed, Jul 22, 2020 at 3:14 PM Wei Zhong wrote: > > Congratulations! Thanks Dian for the great work! > > Best, > Wei > > > 在 2020年7月22日,15:09,Leonard Xu 写道: > > >

Re: flink1.9写权限认证的es6

2020-07-16 文章 Yangze Guo
Hi, SQL添加认证的逻辑已经在FLINK-18361[1] 中完成了,1.12版本会支持这个功能 [1] https://issues.apache.org/jira/browse/FLINK-18361 Best, Yangze Guo On Fri, Jul 17, 2020 at 10:12 AM Dream-底限 wrote: > > hi: > 请问flink如何将数据写入到权限认证的es集群哪,没找到配置用户名密码的地方,哪位大佬帮忙解答一下

Re: Re: flink on yarn日志问题

2020-07-13 文章 Yangze Guo
Hi, 王松 我理解拼接url就可以了,不用实际去登陆机器然后进到对应目录。 Best, Yangze Guo On Tue, Jul 14, 2020 at 8:26 AM 王松 wrote: > > 我们也有问题 1,和 Yangze Guo 说的一样,每次都要去对应的tm目录中去找日志,很麻烦,不知道有没有更简单的办法。 > > Yangze Guo 于2020年7月13日周一 下午5:03写道: > > > 1. > > 我验证了一下,如果开启了日志收集,那tm的日志是会保存的,但是你整个application结束前可

Re: Re: Re: flink 1.11 es未定义pk的sink问题

2020-07-13 文章 Yangze Guo
Best, Yangze Guo On Mon, Jul 13, 2020 at 3:44 PM sunfulin wrote: > > hi,YangZe,Leonard, > 我增加了一个可以复现问题的测试类,可以执行下看看。可以明显观察到,两个sink在有PK时写入正常,在没有PK时只有一条记录(id是索引名)。 > > import org.apache.flink.api.common.typeinfo.Types; > > import org.apache.flink.streaming.api.datastream.D

Re: Re: flink on yarn日志问题

2020-07-13 文章 Yangze Guo
1. 我验证了一下,如果开启了日志收集,那tm的日志是会保存的,但是你整个application结束前可能看不到,有一个trick的方法,首先在jm日志中找到tm分配到了哪个NodeManager上,通过拼接url的方式来获取container的日志 2. 你是否需要调整一下重启策略[1]? 如果开启了ck,默认情况下就会一直尝试重启job [1] https://ci.apache.org/projects/flink/flink-docs-master/zh/dev/task_failure_recovery.html Best, Yangze Guo On Mon

Re: Flink es7 connector认证问题

2020-07-13 文章 Yangze Guo
Hi, 请问您有检查过pt.get("es.user.name"), pt.get("es.user.password")这两个参数读出来是否都是正确的,另外更完整的错误栈方便提供下么? Best, Yangze Guo On Mon, Jul 13, 2020 at 3:10 PM 李宇彬 wrote: > > 各位好, > 请教一个问题 > 我们生产环境的es7是有用户名密码认证的,使用如下代码启动后会报,这段代码调用了es rest client a

Re: Re: flink 1.11 es未定义pk的sink问题

2020-07-13 文章 Yangze Guo
INSERT走的就是processUpsert这个方法,当不指定PK时,生成的key会是null,然后创建一个IndexRequest。 Best, Yangze Guo On Mon, Jul 13, 2020 at 2:00 PM sunfulin wrote: > > > hi, Leonard > 我定义了一个ddl和一个dml,sql如下。ddl中没有定义PK,我观察到的现象是:这样在sink到es结果中,结果生成的id是index名,导致只有一条记录。 > 我将DDL更换为之前版本的with参数(声明使用update-mode = >

Re: flink on yarn日志问题

2020-07-12 文章 Yangze Guo
/projects/flink/flink-docs-master/zh/ops/deployment/yarn_setup.html#run-a-single-flink-job-on-yarn Best, Yangze Guo On Mon, Jul 13, 2020 at 10:49 AM 程龙 <13162790...@163.com> wrote: > > 请问一下两个问题 > 1 flink on yarn的时候 taskmanager 挂掉的时候 上面的日志会被删除掉 无法查看 ,除了使用es收集日志的这种方案, > 还有没有可以使taskman

Re: flink 1.11 es未定义pk的sink问题

2020-07-12 文章 Yangze Guo
/streaming/connectors/elasticsearch/table/RowElasticsearchSinkFunction.java#L102 [2] https://github.com/elastic/elasticsearch/blob/977230a0ce89a55515dc6ef6452e9f059d9356a2/core/src/main/java/org/elasticsearch/action/index/IndexRequest.java#L509 Best, Yangze Guo On Sat, Jul 11, 2020 at 11:33 PM

Re: 作业升级到flink1.11,idea运行失败

2020-07-07 文章 Yangze Guo
尝试加一下这个依赖 groupId: org.apache.flink artifactId: flink-clients_${scala.binary.version} Best, Yangze Guo On Wed, Jul 8, 2020 at 11:27 AM SmileSmile wrote: > > > hi > > 作业的依赖从1.10.1升级到1.11.0,在idea运行的时候报错 > > Exception in thread "main" java.lang.IllegalStateExcepti

Re: flink1.9 on yarn

2020-06-27 文章 Yangze Guo
我理解你需要使用session模式,即./bin/yarn-session.sh [1] [1] https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/yarn_setup.html#flink-yarn-session Best, Yangze Guo On Sun, Jun 28, 2020 at 9:10 AM guanyq wrote: > > 问题1 > > ./bin/flink run -m > yarn-cluster每次提交是

Re: [ANNOUNCE] Yu Li is now part of the Flink PMC

2020-06-16 文章 Yangze Guo
Congrats, Yu! Best, Yangze Guo On Wed, Jun 17, 2020 at 9:35 AM Xintong Song wrote: > > Congratulations Yu, well deserved~! > > Thank you~ > > Xintong Song > > > > On Wed, Jun 17, 2020 at 9:15 AM jincheng sun wrote: >> >> Hi all, >> >> On beh

Re: 关于kubernetes native配置的疑问

2020-05-28 文章 Yangze Guo
您好,我来回来一下第一个大问题 1. resources.requests.cpu和resources.limits.cpu都会被设置为kubernetes.jobmanager.cpu 2. external-resource..kubernetes.config-key 是为1.11的新特性扩展资源框架[1]而加入的。请不要使用它来配置cpu和memory。 [1] https://issues.apache.org/jira/browse/FLINK-17044 Best, Yangze Guo On Thu, May 28, 2020 at 3:48 PM wrote

Re: 向flink push代码

2020-05-27 文章 Yangze Guo
您好,社区的贡献代码教程[1]。 Tips: 如果不是hotfix,请提交前确认已经在ticket下达成一致并且有committer将其assign给您。 [1] https://flink.apache.org/zh/contributing/contribute-code.html Best, Yangze Guo On Thu, May 28, 2020 at 10:15 AM 宇张 wrote: > > 找打了教程了 > > > On Thu, May 28, 2020 at 10:10 AM 宇张 wrote: > >

Re: sql client定义指向elasticsearch索引密码问题

2020-05-21 文章 Yangze Guo
目前1.11已经feature freeze,该功能最早1.12才能支持,着急的话可以看看DataStream API的ElasticSearchSink,这个是支持安全认证的,也可以自己实现一个TableSink Best, Yangze Guo On Fri, May 22, 2020 at 9:59 AM Rui Li wrote: > > Hi,目前还不支持,不过有PR在做这个功能:https://github.com/apache/flink/pull/11822 > > On Wed, May 20, 2020 at 4:10 PM natura

Re: [ANNOUNCE] Apache Flink 1.10.1 released

2020-05-18 文章 Yangze Guo
Thanks Yu for the great job. Congrats everyone who made this release possible. Best, Yangze Guo On Mon, May 18, 2020 at 10:57 AM Leonard Xu wrote: > > > Thanks Yu for being the release manager, and everyone else who made this > possible. > > Best, > Leonard Xu > > 在 2

Re: flink1.10 ES sink 配置输出object字段问题

2020-05-14 文章 Yangze Guo
ES sink 是Json format的,你可以参考 [1] [1] https://ci.apache.org/projects/flink/flink-docs-master/dev/table/connect.html#json-format Best, Yangze Guo On Thu, May 14, 2020 at 3:37 PM XiaChang <13628620...@163.com> wrote: > > hi,大家好 > > > 请问ES index中存在object字段时,flink1.10版本ES sink 的ddl该如何配置。

Re: 在使用flink1.7.2写入ES6的时候,有时会报错:request retries exceeded max retry timeout [30000]

2020-05-13 文章 Yangze Guo
ers(...) restClientBuilder.setMaxRetryTimeoutMillis(...) restClientBuilder.setPathPrefix(...) restClientBuilder.setHttpClientConfigCallback(...) } ); [1] https://ci.apache.org/projects/flink/flink-docs-master/dev/connectors/elasticsearch.html#elasticsearch-sink Best, Yangze Guo On Wed, May 13, 2020 at 2:53 PM Jim C

Re: 请问有没有什么方法可以把checkpoint打到集群外的hdfs?

2020-04-15 文章 Yangze Guo
1. flink会去读YARN_CONF_DIR or HADOOP_CONF_DIR这两个环境变量 2. 我理解这和你flink运行的集群是解耦的,只要你dir的路径不变,就会从那个dir找checkpoint恢复 Best, Yangze Guo On Wed, Apr 15, 2020 at 3:38 PM tao wang wrote: > > 多谢回复, 还有几个问题请教: > 1、外部集群的hdfs-site, core-site 这些怎么配置? > 2、另外一个角度, 如果我把任务迁移到另外一个集群,如何让它从老的集群的checkpoint 恢复

Re: flink java.util.concurrent.TimeoutException

2020-04-15 文章 Yangze Guo
日志上看是Taskmanager心跳超时了,如果tm还在,是不是网络问题呢?尝试把heartbeat.timeout调大一些试试? Best, Yangze Guo On Mon, Apr 13, 2020 at 10:40 AM 欧阳苗 wrote: > > job运行了两天就挂了,然后抛出如下异常,但是taskManager没有挂,其他的job还能正常在上面跑,请问这个问题是什么原因导致的,有什么好的解决办法吗 > > > 2020-04-13 06:20:31.379 ERROR 1 --- [

Re: flink 1.7.2 YARN Session模式提交任务问题求助

2020-04-15 文章 Yangze Guo
Flink需要设置hadoop相关conf位置的环境变量 YARN_CONF_DIR or HADOOP_CONF_DIR [1] [1] https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/yarn_setup.html Best, Yangze Guo On Mon, Apr 13, 2020 at 10:52 PM Chief wrote: > > 大家好 > 目前环境是flink 1.7.2,使用YARN Session模式提交任务,Hadoop 版本2

Re: 请问有没有什么方法可以把checkpoint打到集群外的hdfs?

2020-04-15 文章 Yangze Guo
checkpoint的目录设置key为state.checkpoints.dir 你可以这样设置 state.checkpoints.dir: hdfs://namenode:port/flink/checkpoints/ Best, Yangze Guo On Wed, Apr 15, 2020 at 1:45 PM tao wang wrote: > > 现在有个场景, 集群上上了一些比较重要的任务,为了提高这些任务的性能,为它提供了一个专有的hdfs 集群用来存储checkpoint 。 > > 但是现在不知道怎么配置才能让支持任务打到外部的hdfs集群。 > > 谢谢!!

Re: 从savepoint不能恢复问题

2020-04-02 文章 Yangze Guo
/state/savepoints.html#faq Best, Yangze Guo On Thu, Apr 2, 2020 at 6:01 PM 酷酷的浑蛋 wrote: > > > > Failed to rollback to checkpoint/savepoint > hdfs://xxx/savepoint-9d5b7a-66c0340f6672. Cannot map checkpoint/savepoint > state for operator cbc357ccb763df2852fee8c4fc7d55f2 to the new p

Re: [Third-party Tool] Flink memory calculator

2020-04-01 文章 Yangze Guo
@Marta Thanks for the tip! I'll do that. Best, Yangze Guo On Wed, Apr 1, 2020 at 8:05 PM Marta Paes Moreira wrote: > > Hey, Yangze. > > I'd like to suggest that you submit this tool to Flink Community Pages [1]. > That way it can get more exposure and it'll be easier fo

Re: [Third-party Tool] Flink memory calculator

2020-03-31 文章 Yangze Guo
Hi, there. In the latest version, the calculator supports dynamic options. You could append all your dynamic options to the end of "bin/calculator.sh [-h]". Since "-tm" will be deprecated eventually, please replace it with "-Dtaskmanager.memory.process.size=". Be

Re: [Third-party Tool] Flink memory calculator

2020-03-29 文章 Yangze Guo
deployed in case there is any unexpected configuration. It's a good point that we need to parse the dynamic configs. I prefer to parse the dynamic configs and cli commands in bash instead of adding hook in JobListener. Best, Yangze Guo On Mon, Mar 30, 2020 at 10:32 AM Jeff Zhang wrote: > > Hi

Re: [Third-party Tool] Flink memory calculator

2020-03-29 文章 Yangze Guo
Hi, Yun, I'm sorry that it currently could not handle it. But I think it is a really good idea and that feature would be added to the next version. Best, Yangze Guo On Mon, Mar 30, 2020 at 12:21 AM Yun Tang wrote: > > Very interesting and convenient tool, just a quick question: could thi

[Third-party Tool] Flink memory calculator

2020-03-27 文章 Yangze Guo
. For more details, please take a look at the README. Any feedback or suggestion is welcomed! [1] https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup.html [2] https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html Best, Yangze Guo

Re: 申请中文社区

2020-03-27 文章 Yangze Guo
您好,如果想订阅user-zh,可以发送邮件到user-zh-subscr...@flink.apache.org Best, Yangze Guo On Fri, Mar 27, 2020 at 4:45 PM 大数据开发面试_夏永权 wrote: > > 申请中文社区,不知是否成功,我非常想加入社区,非常感谢。

Re: 全量编译报错

2020-03-16 文章 Yangze Guo
报错信息是什么呢?代码分支是哪个? 您可以再pull一下最新的master,目前看最新的master是可以全量编译的[1] [1] https://travis-ci.org/github/apache/flink/builds/662890263?utm_source=github_status_medium=notification Best, Yangze Guo Best, Yangze Guo On Mon, Mar 16, 2020 at 4:48 PM 吴志勇 <1154365...@qq.com> wrote: > > 在项目根目录下执

Re: Flink stop many times

2020-03-12 文章 Yangze Guo
> org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException 从日志上看起来像是HDFS问题引起Sink失败,您检查一下nn/dn相关设置/日志有误异常? Best, Yangze Guo On Sun, Mar 8, 2020 at 2:24 AM 273339930 <273339...@qq.com> wrote: > > Dear all: >我的flink经常挂掉,大约6天挂一次,我找了半天实在无法查出什么原因 >场景:读取kafka写hd

Re: yarn per job 模式这个报错原因是什么?随机出现

2019-12-19 文章 Yangze Guo
0这个端口打不开,可以尝试配置一下blob.server.port Best, Yangze Guo On Fri, Dec 20, 2019 at 3:05 PM rockey...@163.com wrote: > > 嗨,大家好,flink per job模式下随机出现如下错误?有知道是什么原因吗? > > 2019-12-20 12:07:03,462 INFO > org.apache.flink.runtime.entrypoint.Cl

Re: Re: flink on yarn 指定节点开启 yarn session 报错

2019-11-26 文章 Yangze Guo
8082我记得是historyserver.web.port的默认值,很可能是冲突了改成8081呢? Best, Yangze Guo On Wed, Nov 27, 2019 at 11:13 AM 杨浩程 wrote: > > > 好的

Re: flink on yarn 指定节点开启 yarn session 报错

2019-11-26 文章 Yangze Guo
您好,目前ML不支持图片,能将配置贴上来么? 另外看报错信息像是8082端口被占用导致rest服务起不起来 Best, Yangze Guo On Wed, Nov 27, 2019 at 10:56 AM 杨浩程 wrote: > 各位大佬好! >请教各位个问题:使用的flink 版本1.9.0。测试flink on yarn 指定节点 开启yarn session 会话。 >希望yarnsession开启的集群 jobmanager开启在我指定的节点上。 >更改的配置

Re: taskmanager faild

2019-04-26 文章 Yangze Guo
Hi, 目前apache邮件列表不支持附件[1] 可将问题提至jira[2]或提供图片链接 [1] https://commons.apache.org/mail-lists.html [2] https://issues.apache.org/jira/projects/FLINK/summary Best, Yangze Guo Best, Yangze Guo On Fri, Apr 26, 2019 at 11:01 AM Xintong Song wrote: > > hi naisili, > > 我没有在你的邮件里看到任何附件、截图或者文