????????????????????????????????????????????????checkpoint??????

2021-05-17 Thread 5599
--  -- ??: ""<62...@163.com>; : 2021??5??17??(??) 6:23 ??: "user-zh"

??????????????checkpoint??????????????????????????

2021-10-28 Thread claylin
checkpointflink1.3/1.4+sql??checkpoint1??rocksdb15??30

?????? ??????????????checkpoint??????????????????????????

2021-10-31 Thread claylin
continue trigger --  -- ??: "user-zh"

checkpoint??????????

2021-11-07 Thread ??????
: flink on yarn ??flink hdfs,ark1??hdfs??active??standby ark2standbyactive :??flink??checkpoint??hdfs??url??hdfs:ark:8082  ,standby??, ??  

checkpoint??????????

2021-11-08 Thread ??????
: flink on yarn ??flink hdfs,ark1??hdfs??active??standby ark2standbyactive :??flink??checkpoint??hdfs??url??hdfs:ark:8082  ,standby??, hdfs??mycluster

????????????checkpoint????

2020-08-21 Thread Robert.Zhang
Hello all, iterative stream job checkpoint??checkpoint state k??org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint tolerable failure threshold

??????????????????checkpoint????

2020-08-25 Thread Robert.Zhang
Hi Congxian, iteration source??barrier?? ??barrier??operator??barrier??checkpoint?? ??operator

?????? ????????????checkpoint????

2020-08-26 Thread Robert.Zhang
6:31?? > > > > &gt; Hello all, > > &gt; iterative stream job > > &gt; checkpoint??checkpoint > > &gt; state > ????k??org.apache.flink.util.FlinkRuntimeExce

?????? ?????? ????????????checkpoint????

2020-08-27 Thread Robert.Zhang
1964 > &gt; &gt; Best, > &gt; &gt; Congxian > &gt; &gt; > &gt; &gt; > &gt; &gt; Robert.Zhang <173603...@qq.com&amp;gt; ??2020??8??21?? 6:31?? > &gt; &gt; > &gt; &gt; &amp;gt; Hello all, > &gt; &

????????????checkpoint????????????

2020-09-08 Thread ????
Hi RocksDB Checkpoint Checkpoint Checkpoint RocksDB ?? Compaction Checkpoint ?? Checkpoint

????????????checkpoint????????????

2020-09-08 Thread superainbower
| ?? ??2020??09??8?? 19:56??<836961...@qq.com> ?? Hi RocksDB Checkpoint ???? Checkpoint ???? Checkpoint RocksDB ?? Compaction ???? Chec

????????????checkpoint????????????

2020-09-08 Thread superainbower
; ??savepoint flink1.11.1 | | | | | superainbower | | superainbo...@163.com | ?? ??2020??09??8?? 19:56??<836961...@qq.com> ?? Hi RocksDB Checkpoint ???? Checkpoint ???? Chec

checkpoint??????????????

2019-11-27 Thread sun
??       flink ??checkpoint?? ??

??checkpoint????????????

2020-01-16 Thread claylin
1.9.1checkpointsst?? 2020-01-16 19:29:39 java.lang.Exception: Exception while creating StreamOperatorStateContext. at

??????????checkpoint

2020-01-16 Thread sun
??checkpointcheckpoint??

?????? ??????????checkpoint

2020-01-16 Thread sun
?? env.enableCheckpointing(CommonConstants.CHECK_POINTING); env.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime); env.setParallelism(CommonConstants.PARALLELISM_NUMBER); // env.setRestartStrategy(RestartStrategies.noRestart()); /

?????? ?????? ??????????checkpoint

2020-01-17 Thread sun
--  -- ??: "Yun Tang"https://aka.ms/ghei36>; From: sun <1392427...@qq.com> Sent: Friday, January 17, 2020 1:43:53 PM To: user-zh

flink ????checkpoint????

2021-12-27 Thread ??????????
:flink1.14.0 : flink 1.14.0??jdk1.7??yarn??checkpoint,IN_PROGRESS : ./bin/flinksql-submit.sh \ --sql sqlserver-cdc-to-kafka.sql \ -m yarn-cluster \ -ynm sqlserverTOkafka \ -ys 2 \ -yjm 1024 \ -ytm 1024 \ -yid

Unaligned Checkpoint

2022-06-11 Thread 小昌同学
大佬们可以说说Unaligned Checkpoint的实现吗 看了不少文档 没有太看懂 我如果想在sql里面实现 这个该怎么设置啊 请大佬们指教 | | 小昌同学 | | ccc0606fight...@163.com |

????flink??????????????????checkpoint????

2022-06-23 Thread ??????
flink??1.13.1 hdfs??3+ ?? 2022-06-24 10:58:19,839 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator[] - Decline checkpoint 1101 by task b3d88f9ef72bda003056856c4422742d of job 6bd7dc46451f01e008762c9b556cb08f at zhaohy4-test-taskmanager-1-1 @ 10.42.5.55

state??????checkpoint??????

2020-07-16 Thread sun
env.enableCheckpointing(1000);env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime); // env.setRestartStrategy(RestartStrategies.noRestart()); env.getCheckpointConfig().setCheckpointTimeout(500); env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTL

?????? state??????checkpoint??????

2020-07-17 Thread sun
??counts ?? Listhttps://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/libs/state_processor_api.html Best, Congxian sun <1392427...@qq.com> ??2020??7??16?? 6:16?? > > env.enableCheckpointing(1000);env.setStreamTimeCharacteristic(TimeCha

??????state??????checkpoint??????

2020-07-17 Thread JasonLee
hi UI??checkpoint??checkpoint?? ?? | | JasonLee | | ??17610775...@163.com | Signature is customized by Netease Mail Master ??2020??07??17?? 17:21??sun ?? ??counts ?? Listhttps://ci.apache.org/projects

?????? state??????checkpoint??????

2020-07-20 Thread sun
JM checkpoint ?? 18:08:07.615 [Checkpoint Timer] INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator  - Triggering checkpoint 116 @ 1595239687615 for job acd456ff6f2f9f59ee89b126503c20f0. 18:08:07.628 [flink-akka.actor.default-dispatcher-420] INFO

??????checkpoint??????state

2020-09-03 Thread sun
??2?? 1checkpointchk-  ??chk-1??chk-2 , 2checkpoint??state?? ?? final StreamExecutionEnvironment streamExecutionEnvironment

??????checkpoint??????state

2020-09-03 Thread sun
??2?? 1checkpointchk-  ??chk-1??chk-2 , 2checkpoint??state?? ?? final StreamExecutionEnvironment streamExecutionEnvironment

??????checkpoint??????state

2020-09-03 Thread sun
??2?? 1checkpointchk-  ??chk-1??chk-2 , 2checkpoint??state?? ?? final StreamExecutionEnvironment streamExecutionEnvironment

????????????checkpoint??????state

2020-09-03 Thread sun
--  -- ??: "user-zh" <13162790...@163.com

??????checkpoint??????state

2020-09-03 Thread sun
??2?? 1checkpointchk-  ??chk-1??chk-2 , 2checkpoint??state?? ?? final StreamExecutionEnvironment streamExecutionEnvironment

flunk - checkpoint

2020-09-27 Thread 郝文强
checkpoint 到底是在什么时间点触发的呢? 我看过官方关于checkpoint的文档,但还是说不清 checkpoint 是什么时候触发的求大佬指点 | | 郝文强 | | 18846086...@163.com | 签名由网易邮箱大师定制

Re:flunk - checkpoint

2020-09-28 Thread hailongwang
Source 算子的Checkpoint 是由 CheckpointCoordinator 周期性触发的(时间周期是自己设置的参数),然后下面的算子等到收到上游所有并发的barrier时,触发自己的Checkpoint(PS:1.11 目前 实现了 unaligned Checkpoint,无需对齐)。当Master 收到所有算子的 Checkpoint 完成后,会 notifyCheckpointComplete。 希望对你有帮助。 祝好。 Hailong Wang 在 2020-09-28 09:34:17,"郝文强" <18846086..

checkpoint、state

2019-11-29 Thread hahaha sc
flink的每条数据既然都做了checkpoint,做成全局分布式一致性快照,那还需要本地state干啥呢?是否可以理解成,本地state是一致性快照的一部分而已? 昨天看了 社区的直播回放,听PMC的介绍,好像不是一回事。

????checkpoint ????190G??

2019-11-29 Thread sun
?? iPhone

??????????checkpoint ????190G??

2019-11-29 Thread sun
-- -- ??: "Henry" : 2019??11??30??(??) 10:25 ??: "user-zh"; ????: Re:checkpoint 190G?? ?? 2019-11-30 10:13:27??&quo

?????? ????checkpoint ????190G??

2019-11-29 Thread sun
retain ??20??increase?? -- -- ??: "tison" : 2019??11??30??(??) 10:31 ??: "user-zh"; : Re: checkpoint 190G?? retain

????: ????checkpoint ????190G??

2019-11-29 Thread tangjunli...@huitongjy.com
backend?? state.backend.incremental ??? tangjunli...@huitongjy.com sun ?? 2019-11-30 10:13 flink; user-zh-subscribe ?? checkpoint 190G?? ?? iPhone

?????? ????checkpoint ????190G??

2019-11-29 Thread sun
rocksdb true-- -- ??: "tangjunli...@huitongjy.com" : 2019??11??30??(??) 2:03 ??: "user-zh"; : : checkpoint 190G?? backend?? state.backend.incremental

????: ?????? ????checkpoint ????190G??

2019-11-29 Thread tangjunli...@huitongjy.com
??state.backend.incremental false tangjunli...@huitongjy.com sun ?? 2019-11-30 14:05 user-zh ?? ?? checkpoint 190G?? rocksdb true-- -- ??: "tang

?????? ?????? ????checkpoint ????190G??

2019-11-29 Thread sun
??checkpoint checkpoint -- -- ??: "tangjunli...@huitongjy.com" : 2019??11??30??(??) 2:06 ??: "user-zh"; ????: : ?? ???

????: ?????? ????checkpoint ????190G??

2019-11-29 Thread tangjunli...@huitongjy.com
??state.backend.incremental??false??checkpointflinkTTL tangjunli...@huitongjy.com sun ?? 2019-11-30 14:08 user-zh ?? ?? ?? checkpoint 190G

?????? ?????? ????checkpoint ????190G??

2019-11-29 Thread sun
??-- -- ??: "tangjunli...@huitongjy.com" : 2019??11??30??(??) 2:11 ??: "user-zh"; ????: : ?? checkpoint 190G?? ??state.backend.incremental??false

?????? ?????? ????checkpoint ????190G??

2019-11-29 Thread sun
??-- -- ??: "tison" : 2019??11??30??(??) 2:12 ??: "user-zh"; ????: Re: ?? checkpoint 190G?? retain ??1 2 3 ... Best, tison. su

????: ?????? ????checkpoint ????190G??

2019-11-30 Thread 2421307...@qq.com
hi ?? 2421307...@qq.com sun ?? 2019-11-30 14:15 user-zh ?? ?? ?? checkpoint 190G?? ??-- -- ??: "tison" : 20

FlinkSQL????????????checkpoint????

2020-04-15 Thread Even
Hi?? ?? 1?? Flink SQL CLI ??DDL??checkpoint?? 2?? Flink SQL CLI ??CLItable??

flink做checkpoint失败 Checkpoint Coordinator is suspending.

2021-02-01 Thread chen310
flink做checkpoint一直失败,请教下是啥原因 <http://apache-flink.147419.n8.nabble.com/file/t572/config.config> <http://apache-flink.147419.n8.nabble.com/file/t572/history.history> <http://apache-flink.147419.n8.nabble.com/file/t572/history_detail.history_detail> -- Sent from: h

oss checkpoint fail

2022-05-12 Thread json
使用oss 存储checkpoint,做几次checkpoint就会出现下面报错,导致checkpoint失败 Caused by: org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss.ClientException: Connection error due to: Trying to access closed classloader. Please check if you store classloaders directly or indirectly in static fields. If the

Re: Unaligned Checkpoint

2022-06-12 Thread Zhanghao Chen
你好, Unaligned checkpoint 是个底层特性,要使用的话只要设置 Flink 参数 execution.checkpointing.unaligned = true 就行,在 SQL client 中,可以使用 SET "key" = "value" 的语法设置 Flink 参数的值。 Unaligned checkpoint 较之 aligned checkpoint 主要的改变在于 * unaligned cp 在输入缓冲区收到第一个 cp barrier 的时候立即触发快照并直接输出至下游;代价是快照需要

Flink ??kafka??????????????????checkpoint??????????

2020-07-20 Thread ??????
>>>>?? >>>>??Flink ??kafka??????checkpointEXACTLY_ONCE >>?? Producer attempted an operation with an old epoch.Either there is a newer producer with the same transactionalId, or the producer's transaction has been expired by the broker >>?? >>??

??????????????Flink????FINISHED??????????????Checkpoint??????

2020-08-14 Thread ????
?? redisredis??State --  -- ??:

checkpoint问题

2020-09-16 Thread smq
如果我的程序逻辑修改,还能用之前的checkpoint吗

Re: flunk - checkpoint

2020-09-27 Thread Congxian Qiu
Hi checkpoint 会按照用户设定的周期定期触发, 同时也会收到 minPauseBetweenCheckpoints 以及 maxConcurrentCheckpoints 等参数的控制,具体的可以看一下这个文档[1], [1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/stream/state/checkpointing.html#enabling-and-configuring-checkpointing Best, Congxian 郝文强 <18846086...@

?????? Flink 1.10.1 checkpoint????????

2020-10-09 Thread ??????
jdk8jdk261?? --  -- ??: "user-zh"

??????????: ??????flink-1.11.2 ????checkpoint????

2020-11-12 Thread ?n?e?m?a ????????
??1.11.2??debug?? ?? --  -- ??:

[State Backend] ????????????checkpoint????????

2019-05-02 Thread eric
??: ??flink, ??state checkpoint??: 1) socketkeyed state backend; job?? 2) socket??job??failed 3) socket 4) flink??socket, ??job??, ??MapState

Checkpoint使用

2019-08-27 Thread yanggang_it_job
关于flink从checkpoint的问题: 1、如果我的并行度发生了改变,怎么从checkpoint启动? 2、是否可以动态设置checkpoint触发时间?

????????????state??????checkpoint size????????????????????

2019-08-29 Thread Bo?????WJay
state?? ??checkpoint size??8checkpoint size??1.6??Gstate?? jobprocessFunction val stream = env.addSource

Re: checkpoint、state

2019-12-01 Thread Congxian Qiu
Hi Checkpoint 是所有本地 state 的一个快照,用于程序出现故障后,进行恢复使用。 Best, Congxian hahaha sc 于2019年11月29日周五 下午4:12写道: > > flink的每条数据既然都做了checkpoint,做成全局分布式一致性快照,那还需要本地state干啥呢?是否可以理解成,本地state是一致性快照的一部分而已? > 昨天看了 社区的直播回放,听PMC的介绍,好像不是一回事。 >

flink savepoint checkpoint

2019-12-10 Thread lucas.wu
hi 各位: 有个问题想和大家讨论一下,就是flink的savepoint为什么要设置成手动的?如果在stop程序的时候没有做savepoint的话,在我重启的时候就不能使用之前保存的一些状态信息。为什么不参考spark的方式,定时做checkpoint,然后启动的时候指定ck地址就可以从上次执行的地方继续执行。

Re: checkpoint、state

2020-01-09 Thread Px New
Yes, CheckPoint 是一个封装后的Meta信息 而这个被封装的Meta信息是有所有Operator 的state 的组成了 hahaha sc 于2019年11月29日周五 下午4:12写道: > > flink的每条数据既然都做了checkpoint,做成全局分布式一致性快照,那还需要本地state干啥呢?是否可以理解成,本地state是一致性快照的一部分而已? > 昨天看了 社区的直播回放,听PMC的介绍,好像不是一回事。 >

Re: checkpoint、state

2020-01-12 Thread LakeShen
状态先存到本地 比直接存到 HDFS 速度更快,对于 RocksDBStateBackend,状态会先存到本地,然后异步存储到 HDFS。 Px New <15701181132mr@gmail.com> 于2020年1月10日周五 上午10:27写道: > Yes, CheckPoint 是一个封装后的Meta信息 而这个被封装的Meta信息是有所有Operator 的state 的组成了 > > hahaha sc 于2019年11月29日周五 下午4:12写道: > > > > > > flink的每条数

????checkpoint??hdfs??kerberos????????????

2020-04-13 Thread ??????
org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 288 @ 1586834050629 for job 45e07f380ba6607cb93b01bbbcd45729. 2020-04-14 11:14:40,936 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 288 for job 45e07f380ba6607cb93b01bbbcd45729 (3705 bytes in 30224 ms). 2020-04-14 11

Re: flink做checkpoint失败 Checkpoint Coordinator is suspending.

2021-02-01 Thread chen310
org.apache.flink.runtime.checkpoint.CheckpointCoordinator[] - Triggering checkpoint 8697 (type=CHECKPOINT) @ 1612169690917 for job 1299f2f27e56ec36a4e0ffd3472ad399. 2021-02-01 08:54:50,999 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator[] - Decline checkpoint 8697 by task

Re: flink做checkpoint失败 Checkpoint Coordinator is suspending.

2021-02-02 Thread Congxian Qiu
Hi 你 flink 是什么版本,以及你作业 checkpoint/state 相关的配置是什么呢?如果可以的话,把完整的 jm log 发一下 Best, Congxian chen310 <1...@163.com> 于2021年2月1日周一 下午5:41写道: > 补充下,jobmanager日志异常: > > 2021-02-01 08:54:43,639 ERROR > org.apache.flink.runtime.rest.handler.job.JobDetailsHandler [] - Exception

Re: flink做checkpoint失败 Checkpoint Coordinator is suspending.

2021-02-02 Thread chen310
flink版本是1.11,checkpoint配置是: pipeline.time-characteristic EventTime execution.checkpointing.interval 60 execution.checkpointing.min-pause 12 execution.checkpointing.timeout 12 execution.checkpointing.externalized-checkpoint-retention RETAIN_ON_CANCELLATION state.backend rocksdb

Checkpoint Aligned问题

2021-03-30 Thread 张韩

??????????????????UV??????????????MapState??BloomFilter,??checkpoint????????????????????

2020-08-26 Thread x
UV??MapStateBloomFilter??,checkpoint??bloomMapState

flink-sql 1.11??????????????????checkpoint??

2020-09-08 Thread ??????
enableCheckpointing??checkpoint??

?????? flink-sql 1.11??????????????????checkpoint??

2020-09-08 Thread ??????
savapoint??cancel?? --  -- ??: "user-zh"

Re: checkpoint问题

2020-09-16 Thread Yun Tang
Hi checkpoint使用operator id进行一一映射进行恢复,请参照 设置id[1],以及如果checkpoint中存在某个operator但是修改后的作业并不存在该operator时的处理逻辑[2] [1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/state/savepoints.html#assigning-operator-ids [2] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops

????????checkpoint????????23????????????????????????????kafka?????????????????????????? ??????????23??????????????

2020-10-26 Thread sun
checkpoint23kafka?? ??23?? nohup /opt/flink-1.10.2/bin/flink run -c com.toonyoo.app.data.realcomputation.Application /home/test/ty-app-data-real-computation-1.0.jar -s /opt/flink-1.10.2/checkpoints

flink checkpoint kafka CommitFailedException

2019-07-03 Thread ??????
flink1.4.2 kafka 0.10.2 checkppintwarn kafka?? Properties properties = new Properties(); properties.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,kafkaServer); properties.setProperty(ConsumerConfig.GROUP_ID_CONFIG,"user-compute-test1"); properties.setProperty(ConsumerConfig.E

Re: Checkpoint使用

2019-08-27 Thread Congxian Qiu
Hi 1. 你可以按照文档[1]从一个 retained checkpoint 进行恢复,并发度可以改变,但是你需要保证最大并发是一样的 2. 不能动态设置 checkpoint 触发时间 [1] https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/state/checkpoints.html#resuming-from-a-retained-checkpoint Best, Congxian yanggang_it_job 于2019年8月27日周二 下午6:08写道: > 关于fli

Fwd: Checkpoint使用

2019-08-27 Thread Andrew Lin
1,你说的应该是savepoint吧,checkpoint是运行中failover自动恢复的,savepoint如果设置了uid,改变并行度是可以从savepoint启动的 > 下面是被转发的邮件: > > 发件人: yanggang_it_job > 主题: Checkpoint使用 > 日期: 2019年8月27日 GMT+8 下午6:08:24 > 收件人: user-zh@flink.apache.org > 回复-收件人: user-zh@flink.apache.org > > 关于flink从checkpoin

Re:Fwd: Checkpoint使用

2019-08-27 Thread yanggang_it_job
您说的也对,但是我们重启的时候也是可以从checkpoint启动的。当我们安装flink的时候可以指定checkpoint的保存地址,同时我们编写任务的时候设置checkpoint的生成规则,当我们需要重启任务的时候直接指定这个地址就可以了。相对于savepoint来说checkpoint是自动生成的同时更轻量。所以我们现在是基于checkpoint重启的,然后就遇到了我提的那个问题。 在 2019-08-27 19:57:37,"Andrew Lin" 写道: >1,你说的应该是savepoint吧,checkpoint是运行中failover自动恢复的,s

Re: flink savepoint checkpoint

2019-12-10 Thread Yun Tang
Hi Checkpoint 是自动的,你可以配置retain checkpoint[1] 然后从checkpoint 恢复[2],可以不需要一定触发Savepoint。 [1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/checkpoints.html#retained-checkpoints [2] https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/checkpoints.html#resuming-from-a

Re: flink savepoint checkpoint

2019-12-11 Thread 陈帅
flink 1.9里面支持cancel job with savepoint功能 https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/savepoints.html#cancel-job-with-savepoint checkpoint可能是增量的,但savepoint是全量的。具体区别可以参考 https://www.ververica.com/blog/differences-between-savepoints-and-checkpoints-in-flink lucas.wu 于2019年12月

Re: flink savepoint checkpoint

2019-12-12 Thread Congxian Qiu
Flink 也支持从 retained checkpoint 进行恢复,可以参考文档[1] [1] https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/state/checkpoints.html#resuming-from-a-retained-checkpoint Best, Congxian 陈帅 于2019年12月11日周三 下午9:34写道: > flink 1.9里面支持cancel job with savepoint功能 > > https://ci.apache.org

Re: flink savepoint checkpoint

2020-01-09 Thread Px New
stop程序的时候没有做savepoint的话,在我重启的时候就不能使用之前保存的一些状态信息。为什么不参考spark的方式,定时做checkpoint,然后启动的时候指定ck地址就可以从上次执行的地方继续执行。

Re: flink savepoint checkpoint

2020-01-09 Thread zhisheng
hi,这个参数我理解的作用应该是:作业取消的时候是否清除之前的 checkpoint,但这个 checkpoint 的并不一定是作业最新的状态。而如果你取消命令的时候触发一次 savepoint,那么这次的状态就是最新且最全的。 祝好! zhisheng Px New <15701181132mr@gmail.com> 于2020年1月10日周五 上午10:58写道: > Hello ,针对于你这个问题 在FLink 中 是有这个配置的. 当程序stop时,会额外保存检查点 > --> > > >

Re: flink savepoint checkpoint

2020-01-10 Thread Congxian Qiu
Hi 从 Flink 的角度看,Checkpoint 用户 Job 运行过程中发生 failover 进行恢复,savepoint 用于 Job 之间的状态复用。 另外,从 1.9 开始,可以尝试下 StopWithSavepoint[1],以及社区另外一个 issue 尝试做 StopWithCheckpoint[2] [1] https://issues.apache.org/jira/browse/FLINK-11458 [2] https://issues.apache.org/jira/browse/FLINK-12619 Best, Congxian zhisheng

??????Flink-1.10.0 source??checkpoint??????????????

2020-06-29 Thread Yichao Yang
Hi checkpoint??cpugc?? Best, Yichao Yang --  -- ??: "Tia

flink-checkpoint 问题

2024-01-10 Thread 吴先生
Flink版本: 1.12 checkpoint配置:hdfs 现象:作业由于一些因素第N个checkpoint失败,导致任务重试,任务重试失败,hdfs中不存在第N个chk路径,但是为什么会出现一个第N+1的chk路径,且这个路径下是空的

Re:flink-checkpoint 问题

2024-01-10 Thread ouywl
我记得flink低版本有这个bug,会错误的删除某一个checkpoint的,你这个版本太老了,可以升级到新版本。 The following is the content of the forwarded email From:"吴先生" <15951914...@163.com> To:user-zh Date:2024-01-10 17:54:42 Subject:fli

Flink 1.6, increment Checkpoint, the shared dir stored the last year checkpoint state

2020-01-19 Thread LakeShen
Hi community, now I have a flink sql job, and I set the flink sql sate retention time, there are three dir in flink checkpoint dir : 1. chk -xx dir 2. shared dir 3. taskowned dir I find the shared dir store the last year checkpoint state,the only reason I thought is that the latest

flink job自动checkpoint是成功,手动checkpoint失败

2020-06-19 Thread Zhou Zach
2020-06-19 15:11:18,361 INFO org.apache.flink.client.cli.CliFrontend - Triggering savepoint for job e229c76e6a1b43142cb4272523102ed1. 2020-06-19 15:11:18,378 INFO org.apache.flink.client.cli.CliFrontend - Waiting for response... 2020-06-19 15:11:4

Exception: Could not perform checkpoint

2021-05-16 Thread gen
Hi, all: 我的一个任务 稳定运行了一段时间后, 出现异常 导致不断重启。 版本 Flink 1.10, 请问下 从这个异常上看是业务的问题么,还是flink问题? flink ui上日志为: 2021-05-14 16:12:30 java.lang.Exception: Could not perform checkpoint 6713 for operator Source: kafka topic[mt_elk_log] source -> Flat Map -> Sink: customerAccessLogIndex sink (2/4).

flink 不触发checkpoint

2022-02-16 Thread 董少杰
flink读取csv文件建表,同时消费kafka数据建表,两张表join之后写入hdfs(hudi),读取csv数据的任务已经是finished状态,就会触发不了checkpoint,看有什么办法能让它正常触发checkpoint? flink版本1.12.2。 谢谢! | | 董少杰 | | eric21...@163.com |

Re:flink 不触发checkpoint

2022-02-18 Thread RS
1. 图片挂了,看不到,尽量用文字,或者用图床等工具 2. 启动任务有配置checkpoint吗? 在 2022-02-17 11:40:04,"董少杰" 写道: flink读取csv文件建表,同时消费kafka数据建表,两张表join之后写入hdfs(hudi),读取csv数据的任务已经是finished状态,就会触发不了checkpoint,看有什么办法能让它正常触发checkpoint? flink版本1.12.2。 谢谢! | | 董少杰 | | eric21...@163.com |

flink 1.11 checkpoint使用

2020-07-16 Thread 曹武
我在使用flink 1.11.0中得ddl 部分 采用debezium-json做cdc得时候 从checkpoint恢复以后,新来op=d的数据会删除失败 重启命令:./bin/flink run -m yarn-cluster /root/bigdata-flink-1.0.jar -s hdfs://prehadoop01:8020/flink/checkpoints/4cc5df8b96e90c1c2a4d3719a77f51d1/chk-819/_metadata 代码: EnvironmentSettings settings

Re?? ??????????????????UV??????????????MapState??BloomFilter,??checkpoint????????????????????

2020-08-30 Thread Yichao Yang
om> ??2020??8??27?? 1:48?? > > UV??MapStateBloomFilter??,checkpoint??bloomMapState

sql-client checkpoint sql-client

2020-09-04 Thread 引领
想尝试采用flink-cdc 来清洗数据,但是尝试下来,困于几点: ① 使用sql-client 开启checkpoint ,如果程序挂掉,该如何接着checkpoint,继续执行相应程序。尤其是在执行group by或者是count等操作时该如何办? ② 如果以上方式不行,是否可以采用写代码的形式,重启时指定checkpoint,但还是采用flink-cdc的方式去消费 | | 引领 | | yrx73...@163.com | 签名由网易邮箱大师定制

回复: checkpoint问题

2020-09-16 Thread 明启 孙
感谢解答 smq 发件人: Yun Tang 发送时间: 2020年9月17日 10:30 收件人: user-zh 主题: Re: checkpoint问题 Hi checkpoint使用operator id进行一一映射进行恢复,请参照 设置id[1],以及如果checkpoint中存在某个operator但是修改后的作业并不存在该operator时的处理逻辑[2] [1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/state/savepoints.html#assigning

Checkpoint size的问题

2020-10-28 Thread gsralex
Hi, All Checkpoint 一般Web UI显示的是400MB左右,但是查看HDFS实际的大小,不到1MB(_metadata) ,想问下这之间size的偏差为什么这么大?

??????Re:Re: flink-1.10 checkpoint ?????? NullPointerException

2020-11-03 Thread ??????(Jiacheng Jiang)
 kafka?? wordcount?? kafka??checkpoint ?? | Checkpointing Mode | Exactly Once | | Interval | 5s | | Timeout | 10m 0s | | Minimum Pause Between Checkpoints | 0ms | | Maximum Concurrent Checkpoints | 1 | ???? 5377 ?? checkpoint  NPE

Flink DataStream ElasticsearchSink Checkpoint Error

2019-10-22 Thread 王佩
Flink 写入 Elasticsearch,Checkpoint 一直处于IN_PROGRESS 状态,从而导致数据不能写入ES。 如图: [image: image.png] 帮忙看下! 感谢!

Checkpoint failed all the time

2019-11-03 Thread sllence
Checkpoint will fail for some reasons, Is there a way to make checkpoint fail a certain number of times and then the job will fail automatically

Re: Re: flink savepoint checkpoint

2020-01-10 Thread amen...@163.com
hi,了解到使用stop进行任务停止并触发savepoint,会在停止之前生成max_watermark,并注册event-time计时器,我想请问使用yarn kill方式直接停止任务,会属于cancel还是stop亦或是其他? amen...@163.com From: Congxian Qiu Date: 2020-01-10 17:16 To: user-zh Subject: Re: flink savepoint checkpoint Hi 从 Flink 的角度看,Checkpoint 用户 Job 运行过程中发生 failover 进行恢复,savepoint

Re: Re: flink savepoint checkpoint

2020-01-10 Thread muyexm329
其实这个主要还是要看你checkpoint的时间间隔,就像我们看视频倒退一样,它们是两个不同的后退时间点,savepoint能在当下生成checkpoint数据,但是自动的checkpoint可能还要在更早的时间点上生成checkpoint数据(因为在cancel job的时候可能还不到自动checkpoint时间)。两种都可以,只是是一前一后,这也决定了你任务恢复的快慢。线上需要经常修改的job savepoint很实用。 个人觉得任务失败,不管是哪种方式失败(除非是savepoint),肯定是回到上一个自动checkpoint的点上,不会是在savepoint。 原始邮件

Re: Re: flink savepoint checkpoint

2020-01-10 Thread Px New
t; > From: Congxian Qiu > Date: 2020-01-10 17:16 > To: user-zh > Subject: Re: flink savepoint checkpoint > Hi > 从 Flink 的角度看,Checkpoint 用户 Job 运行过程中发生 failover 进行恢复,savepoint 用于 Job > 之间的状态复用。 > 另外,从 1.9 开始,可以尝试下 StopWithSavepoint[1],以及社区另外一个 issue 尝试做 > StopWithCheckpoi

flink checkpoint不生成

2020-01-13 Thread 起子
各位大佬好: 在执行flink job中发现设置了checkpoint,但是没有checkpoint生成,希望大佬们帮忙指点下,不胜感激,具体描述如下: job 图如下: 算子如下: checkpoint配置如下: 但是结果如下: 没有checkpoint生成 部门 / 数据平台 花名 / 起子 Mobile :159 8810 1848 WeChat :159 8810 1848 Email :q...@dian.so

  1   2   3   4   5   6   7   8   >