ects/flink/flink-docs-release-1.11/ops/config.html#state-backend-rocksdb-memory-managed
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/config.html#state-backend-rocksdb-metrics-block-cache-usage
Best,
Yun Tang
From: Andrey Zagrebin
Sent: Tue
Congratulations , Dian!
Best
Yun Tang
From: Yang Wang
Sent: Friday, August 28, 2020 10:28
To: Arvid Heise
Cc: Benchao Li ; dev ; user-zh
; Dian Fu ; user
Subject: Re: [ANNOUNCE] New PMC member: Dian Fu
Congratulations Dian !
Best,
Yang
Arvid Heise
b.com/dataArtisans/frocksdb/blob/49bc897d5d768026f1eb816d960c1f2383396ef4/include/rocksdb/write_buffer_manager.h#L47
[2]
https://github.com/dataArtisans/frocksdb/blob/49bc897d5d768026f1eb816d960c1f2383396ef4/db/column_family.cc#L196
Best
Yun Tang
From: Juha Mynttinen
Sent: Monday, August 24, 2020
host could be resolved.
You can also check the service of 'jobmanager' whether work as expected via
'kubectl get svc' .
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/ops/upgrading.html#compatibility-table
Best
Yun Tang
From: N
jobmanager high availability[1] and you could refer to [2] for
plans of HighAvailabilityService based on native k8s APIs.
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/ops/jobmanager_high_availability.html
[2] https://issues.apache.org/jira/browse/FLINK-12884
Best
Yun Tang
x27; locates [1] via property {log.file}.
[1]
https://github.com/apache/flink/blob/6b9cdd41743edd24a929074d62a57b84e7b2dd97/flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/BootstrapTools.java#L419
Best,
Yun Tang
From: aj
Sent: Friday, Septem
I think
it's worth to give some hints in Flink documentations.
When talking about your idea to sanity check the arena size, I think a warning
should be enough as Flink seems never throw exception directly when the
performance could be poor.
Best
Yun Tang
/browse/FLINK-14816
Best
Yun Tang
From: Deshpande, Omkar
Sent: Tuesday, September 15, 2020 10:25
To: user@flink.apache.org
Subject: Re: flink checkpoint timeout
I have followed this
https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory
] https://issues.apache.org/jira/browse/FLINK-12470
[4] https://github.com/alibaba/Alink
Best
Yun Tang
From: Cristian Lorenzetto
Sent: Monday, September 14, 2020 18:59
To: user@flink.apache.org
Subject: info about flinkml
Hi i m evaluating to adopt flink instead
Hi Juha
Would you please consider to contribute this back to community? If agreed,
please open a JIRA ticket and we could help review your PR then.
Best
Yun Tang
From: Juha Mynttinen
Sent: Thursday, September 10, 2020 19:05
To: Stephan Ewen
Cc: Yun Tang ; user
://issues.apache.org/jira/browse/FLINK-17800
Best
Yun Tang
From: ירון שני
Sent: Wednesday, September 23, 2020 23:56
To: user@flink.apache.org
Subject: Poor performance with large keys using RocksDB and MapState
Hello,
I have a poor throughput issue, and I think I
lock-Cache#caching-index-filter-and-compression-dictionary-blocks
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/state/state_backends.html#memory-management
Best
Yun Tang
From: ירון שני
Sent: Tuesday, September 29, 2020 17:49
To: Yun Tang
ink-runtime/src/main/java/org/apache/flink/runtime/state/filesystem/AbstractFsCheckpointStorage.java#L60
Best
Yun Tang
From: 大森林
Sent: Saturday, October 3, 2020 9:30
To: David Anderson
Cc: user
Subject: 回复： need help about "incremental checkpoint",T
Hi Song
Flink-1.4.2 is a bit too old, and I think this error is caused by FLINK-8876
[1][2] which should be fixed after Flink-1.5, please consider to upgrade Flink
version.
[1] https://issues.apache.org/jira/browse/FLINK-8876
[2] https://issues.apache.org/jira/browse/FLINK-8836
Best
Yun Tang
hich using the
'TestProcess'? The state might not be restored if you change your code without
id assigned.
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/savepoints.html#assigning-operator-ids
Best
Yun Tang
From: Arpith P
Sent: Tuesd
cess keyed state and timers you have to apply the
ProcessFunction on a keyed stream"?
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/stream/operators/process_function.html#the-processfunction
Best
Yun Tang
From: Arpith P
Sent: Tuesda
200 KB), I think your RocksDB has not
ever triggered compaction to reduce sst files, that's why the size constantly
increase.
Best
Yun Tang
From: sudranga
Sent: Wednesday, October 14, 2020 10:40
To: user@flink.apache.org
Subject: Rocksdb - Incremental vs
[2] https://issues.apache.org/jira/browse/FLINK-17288
Best
Yun Tang
From: Arpith P
Sent: Thursday, October 15, 2020 0:50
To: user
Subject: Large state RocksDb backend increases app start time
Hi,
I'm currently storing around 70GB of data in map sate back
ility service.
[1] https://issues.apache.org/jira/browse/FLINK-11105
[2] https://issues.apache.org/jira/browse/FLINK-12884
Best
Yun Tang
From: Aleksandar Mastilovic
Sent: Thursday, September 26, 2019 1:57
To: Sean Hester
Cc: Hao Sun ; Yuval Itzchakov ; user
Subjec
table/ops/state/checkpoints.html#resuming-from-a-retained-checkpoint
Best
Yun Tang
From: Vishwas Siravara
Sent: Wednesday, October 9, 2019 0:54
To: user
Subject: Flink restoring a job from a checkpoint
Hi guys,
I have a flink streaming job which streams from a kafka sour
Hi Vishwas
This because Flink's checkpoint mechanism could offer you more ability. You
could resume from offset within specific checkpoint instead of last committed
offset not to mention you could benefit from restoring from last timer state,
operator state and keyed state.
Best
Yun
-structured-streaming-SSS-td34776.html#a34779
[5] https://github.com/chermenin/spark-states
[6]
https://docs.databricks.com/spark/latest/structured-streaming/production.html#optimize-performance-of-stateful-streaming-queries
[7] https://issues.apache.org/jira/browse/F
ience. You could contact me in private to get this jar package and
rebuild your Flink runtime to enable write buffer manager future.
[1] https://github.com/dataArtisans/frocksdb/pull/4
[2]
https://github.com/facebook/rocksdb/wiki/Write-Buffer-Manager#limit-total-memory-of-memtables
Bes
not use this?
Best
Yun Tang
From: Congxian Qiu
Sent: Thursday, October 10, 2019 11:52
To: theo.diefent...@scoop-software.de
Cc: user
Subject: Re: Flink restoring a job from a checkpoint
Hi Vishwas
Sorry for the confusing, what Theo said previous is the meaning I
Just a minor supplement @Hao Sun<mailto:ha...@zendesk.com>, if you decided to
drop a operator, don't forget to add --allowNonRestoredState (short: -n) option
[1]
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/savepoints.html#allowing-non-restored-state
Be
important role.
[1] https://flink.apache.org/contributing/code-style-and-quality-preamble.html
[2]
https://ci.apache.org/projects/flink/flink-docs-stable/concepts/programming-model.html#levels-of-abstraction
Best
Yun Tang
From: Deepak Sharma
Sent: Monday
st restore from last
latest checkpoint.
If failing to send record would not cause that sub-task to fail, nothing would
happen and job continues to run but this might be not what you want.
Best
Yun Tang
From: anurag
Sent: Friday, October 11, 2019 2:0
Hi John
The jar is not stored in HA path, I think the answer [1] could help you.
[1]
https://stackoverflow.com/questions/51936608/where-can-i-find-my-jar-on-apache-flink-server-which-i-submitted-using-apache-fl
Best
Yun Tang
From: John Smith
Sent: Friday
your job
from kafka offset before that problematic message.
Best
Yun Tang
From: Flavio Pompermaier
Sent: Friday, October 11, 2019 5:50
To: Yun Tang
Cc: Congxian Qiu ; theo.diefent...@scoop-software.de
; user
Subject: Re: Flink restoring a job from a checkpoint
rely on HA to do job
migration things.
Best
Yun Tang
From: Hao Sun
Sent: Friday, October 11, 2019 8:33
To: Yun Tang
Cc: Vijay Bhaskar ; Yang Wang
; Sean Hester ; Aleksandar
Mastilovic ; Yuval Itzchakov ;
user
Subject: Re: Challenges Deploying Flink With
not
"shuffled" until a window triggered.
[1]
https://github.com/apache/flink/blob/2c2095bdad3d47f27973a585112ed820f457de6f/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/writer/ChannelSelectorRecordWriter.java#L60
Best
Yun Tang
From
-docs-stable/ops/state/checkpoints.html#resuming-from-a-retained-checkpoint
Best
Yun Tang
From: Congxian Qiu
Sent: Friday, October 11, 2019 19:47
To: Flavio Pompermaier
Cc: Yun Tang ; theo.diefent...@scoop-software.de
; user
Subject: Re: Flink restoring a job from a
/blob/2c411686d23f456cdc502abf1c6b97a61070a17d/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java#L307
Best
Yun Tang
From: Steven Nelson
Sent: Wednesday, October 16, 2019 4:31
To: user
Subject: Kinesis Connecto
src/main/java/org/apache/flink/runtime/scheduler/LegacyScheduler.java#L190
Best
Yun Tang
From: Matt Anger
Sent: Thursday, October 17, 2019 5:46
To: user@flink.apache.org
Subject: standalone flink savepoint restoration
Hello everyone,
I am running a flink job in k8s
node.
[1]
Already CC Kurt as he could provide more detail information of this.
[1]
https://github.com/apache/flink/blob/blink/flink-libraries/flink-table/src/main/java/org/apache/flink/table/util/resource/StreamNodeUtil.java#L44
Best
Yun Tang
From: Fanbin Bu
Date: Thursday, October 31, 2019
Hi Srikanth
As RocksDB is a single node DB which just like InfluxDB, I recommend you could
refer to an implementation of InfluxDB sink. [1]
[1] https://github.com/apache/bahir-flink/tree/master/flink-connector-influxdb
Best
Yun Tang
From: OpenInx
Date: Monday, November 4, 2019 at 6:28 PM
To
://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#checkpointing
Best
Yun Tang
From: Simon Su
Date: Tuesday, November 5, 2019 at 10:38 AM
To: dev , user
Subject: Checkpoint in FlinkSQL
Hi All
Does current Flink support to set checkpoint properties while using Flink SQL ?
For example
Hi
首先先判断作业是否在不断地failover，是否有“maximum parallelism”
相关的异常，如果有，说明因为改了并发度而不兼容，实际作业一直都没有从checkpoint正常恢复。
如果作业成功地从checkpoint恢复了，再判断是不是因为task端正在因为正在改并发而导致恢复数据中，如果你的state比较大，这一步骤可能会比较耗时，一般这种情况是source端消费了数据，但是无法向下游发送，整个作业看上去像是一直卡在那边。可以通过task端的jstak看调用栈，看是否有restore相关的栈hang住。
如果以上都不是，那请自行jstack看一下source和下
Best
Yun Tang
On 11/6/19, 5:18 PM, "qq" <471237...@qq.com> wrote:
Hi all.
I want to simulation the shell command which “flink -s savepoint” , this
command only can run with shell command, I want to debug it on dev, local
development environment, anyone could help
]
https://github.com/apache/flink/blob/9b43f13a50848382fbd634081b82509f464e62ca/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/co/BaseBroadcastProcessFunction.java#L50
Best
Yun Tang
From: Komal Mariam
Date: Wednesday, November 6, 2019 at 6:19 PM
To: user
Subject
Yes, just sum all file size within checkpoint meta to get the full checkpoint
size (this would omit some byte stream state handles, but nearly accurate).
BTW, I think user-mail list is the better place for this email-thread, already
sent this mail to user-mail list.
Best
Yun Tang
From: shuwen
Hi Lu
I think RocksDB native metrics [1] could help.
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/config.html#rocksdb-native-metrics
Best
Yun Tang
From: Lu Niu
Date: Friday, November 8, 2019 at 8:18 AM
To: user
Subject: Monitor rocksDB memory usage
Hi,
I read that
Yes, state processor API cannot read window state now, here is the track of
this issue [1]
[1] https://issues.apache.org/jira/browse/FLINK-13095
Best
Yun Tang
From: shuwen zhou
Date: Monday, November 18, 2019 at 12:31 PM
To: user
Subject: Fwd: RocksDB state on HDFS seems not being cleanned
Hi Shuwen
Conceptually, checkpoints in Flink behaves more like a system mechanism to
achieve fault tolerance and transparent for users. On the other hand, savepoint
in Flink behaves more like a user control behavior, can savepoint not satisfy
your demands for crontab?
Best
Yun Tang
From
/flink/flink-docs-stable/ops/state/savepoints.html#what-happens-if-i-delete-an-operator-that-has-state-from-my-job
Best
Yun Tang
From: "min@ubs.com"
Date: Thursday, November 21, 2019 at 5:19 PM
To: "user@flink.apache.org"
Subject: Savepoints and checkpoints
Hi,
Are
-release-1.9/ops/jobmanager_high_availability.html#config-file-flink-confyaml
[2] https://issues.apache.org/jira/browse/FLINK-11105
Best
Yun Tang
From: 曾祥才
Date: Monday, November 25, 2019 at 9:28 AM
To: User-Flink
Subject: flink session cluster ha on k8s
hi, is there any example about ha on k8s
What is the embedded mode mean here? If you refer to SQL embedded mode, you
cannot resume from savepoint now; if you refer to local standalone cluster, you
could use `bin/flink run -s` to resume on a local cluster.
Best
Yun Tang
From: Reo Lei
Date: Tuesday, November 26, 2019 at 12:37 AM
To
/streaming/api/functions/sink/TwoPhaseCommitSinkFunction.java#L324
Best
Yun Tang
From: Congxian Qiu
Date: Monday, December 2, 2019 at 10:41 AM
To: Salva Alcántara
Cc: user
Subject: Re: Using MapState clear, put methods in snapshotState within
KeyedCoProcessFunction, valid or not?
Hi
From
g this, and might cause performance regression as we need more
steps here.
Best
Yun Tang
On 12/2/19, 9:29 PM, "Salva Alcántara" wrote:
Hi Yun,
Thanks for your reply. You mention that
" ‘snapshotState’ and ‘initializeState’ interfaces are used mainly to
snapsh
taskmanager.
[1]
https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/kubernetes.html
[2]
https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/kubernetes.html#session-cluster-resource-definitions
Best
Yun Tang
From: Li Peng
Date: Tuesday, December 10, 2019 at 10:09
Sure, /opt/flink/conf is mounted as a volume from the configmap.
Best
Yun Tang
From: Li Peng
Date: Wednesday, December 11, 2019 at 9:37 AM
To: Yang Wang
Cc: vino yang , user
Subject: Re: Flink on Kubernetes seems to ignore log4j.properties
1. Hey Yun, I'm calling /opt/flink/bin/stand
Hi
Would you please give related code? I think it might due to insufficient hint
to type information.
Best
Yun Tang
From: 杨光
Date: Wednesday, December 11, 2019 at 7:20 PM
To: user
Subject: Scala case class TypeInformation and Serializer
Hi, I'm working on write a flink stream job
/state.html#checkpointedfunction
Best
Yun Tang
From: Gaël Renoux
Sent: Tuesday, December 17, 2019 23:22
To: user
Subject: Restore metrics on broadcast state after restart
Hi everyone
I have an KeyedBroadcastProcessFunction with a broadcast state (a bunch of
previous map state as < 🐿, 1> . If you could
follow this logic, the previous serialize/deserialize of Seq could be
greatly reduced.
Best
Yun Tang
From: Aaron Langford
Sent: Wednesday, December 18, 2019 6:47
To: user@flink.apache.org
Subject: MapState with Li
/runtime/operators/join/stream/StreamingJoinOperator.java#L182
Best
Yun Tang
From: Aaron Langford
Sent: Thursday, December 19, 2019 2:22
To: Yun Tang
Cc: user@flink.apache.org
Subject: Re: MapState with List Type for values
So the suggestion as I read it is to
aming
system should do. If you really need to the exactly once in event-time
processing in this scenario, I suggest to run a batch job later to consume all
data source and use that result as a credible one. For processing-time data,
use Flink to generate a credible result is enough.
Best
Yun
RocksDB folder lies in Flink temporary dir [1] which looks
like flink-io- and the configuration is located in the file named as 'LOG'
with RocksDB directory.
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#io-tmp-dirs
Bes
org/jira/browse/FLINK-13910
Best
Yun Tang
From: tison
Sent: Monday, December 30, 2019 15:44
To: wangl...@geekplus.com.cn
Cc: user
Subject: Re: Submit high version compiled code jar to low version flink cluster?
It possibly fails with incompatibility. Flink doesn
Yun Tang
From: Navneeth Krishnan
Sent: Wednesday, January 8, 2020 12:33
To: user
Subject: Using redis cache in flink
Hi All,
I want to use redis as near far cache to store data which are common across
slots i.e. share data across slots. This data is required
ntime/operators/aggregate/MiniBatchLocalGroupAggFunction.java#L89
From: Navneeth Krishnan
Sent: Wednesday, January 8, 2020 15:36
To: Yun Tang
Cc: user
Subject: Re: Using redis cache in flink
Hi Yun,
Thanks, the way I want to use redis is like a cache not as state ba
flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/operators/aggregate/MiniBatchLocalGroupAggFunction.java#L89
Best
Yun Tang
From: Navneeth Krishnan
Sent: Wednesday, January 8, 2020 15:36
To: Yun Tang
Cc: user
Subject: Re: Using redis cache in
you.
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.9/monitoring/checkpoint_monitoring.html
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.9/monitoring/back_pressure.html
Best
Yun Tang
From: Eva Eva
Sent: Friday, January 10, 2020 10
ueryable_state.html
[2]
https://github.com/apache/flink/blob/7a6ca9c03f67f488e40a114e94c389a5cfb67836/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java#L58
Best
Yun Tang
From: M Singh
Sent: Friday, J
static variable with atomic
reference or synchronization when calling RichFunction#open to initialize and
remember to release resources when calling RichFunction#close .
Best
Yun Tang
From: Navneeth Krishnan
Sent: Monday, January 13, 2020 11:22
To: Yun Tang
Cc
leCheckpointFailureNumber(tolerableDeclinedCheckpointNumber);
Best
Yun Tang
From: jose farfan
Sent: Wednesday, January 15, 2020 23:21
To: ouywl
Cc: user ; user...@flink.apache.org
Subject: Re: When I use flink 1.9.1 and produce data to Kafka 1.1.1, The
streamTask checkpoint err
Congratulations, Dian!
Best
Yun Tang
From: Benchao Li
Sent: Thursday, January 16, 2020 22:27
To: Congxian Qiu
Cc: d...@flink.apache.org ; Jingsong Li
; jincheng sun ; Shuo Cheng
; Xingbo Huang ; Wei Zhong
; Hequn Cheng ; Leonard Xu
; Jeff Zhang ; user
://flink.apache.org/roadmap.html#batch-and-streaming-unification
Best
Yun Tang
From: Soheil Pourbafrani
Sent: Friday, January 17, 2020 1:46
To: user
Subject: Flink Batch mode checkpointing
Hi,
While in Streaming mode I'm using the Flink checkpointing and restart strateg
Hi Stephan,
I am +1 for the change which stores timers in RocksDB by default.
Some users hope the checkpoint could be completed as fast as possible, which
also need the timer stored in RocksDB to not affect the sync part of checkpoint.
Best
Yun Tang
From
removed by checkpoint coordinator but takes too long
to complete before job shut down.
3. This file is still useful. This is possible in theory because some
specific rocksDB sst file might not be selected during compactions for a long
time.
Best
Yun Tang
From
wever, there exists no good
solution for Flink-1.9 currently.
[1]
https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#reporter
Best
Yun Tang
From: Gaurav Singhania
Sent: Monday, January 20, 2020 13:04
To: user@flink.apache.org
Su
Hi David
FlinkKafkaConsumer in itself is RichParallelSourceFunction, and you could call
function below to register your metrics group:
getRuntimeContext().getMetricGroup().addGroup("MyMetrics").counter("myCounter")
Best
Yun Tang
From: D
hat's
because your processing logic has some problem to stuck. On the other hand,
since processing checkpoint and records hold the same lock, we cannot process
checkpoint when the record processing logic did not release the lock.
Best
Yun Tang
From: Alexey Treni
Hi Yi
Can the official doc of writing broad cast state [1] satisfies your request?
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html#broadcast-state-1
Best
Yun Tang
From: Jin Yi
Sent: Thursday, January 23, 2020 8:12
che/flink/streaming/api/operators/co/CoBroadcastWithKeyedOperator.java#L101
Best
Yun Tang
From: Jin Yi
Sent: Monday, January 27, 2020 14:50
To: Yun Tang
Cc: user ; user...@flink.apache.org
Subject: Re: [State Processor API] how to convert savepoint back to broadcast
state
checkpoint, even you could
avoid to record too many logs, and I don't think current checkpoint
configuration is appropriate.
Best
Yun Tang
From: Ahmad Hassan
Sent: Monday, January 27, 2020 20:22
To: user
Subject: Re: Flink RocksDB logs filling up disk
Hi Alexey
If possible, I think you could move some RDBMS maintenance operations to the
#open method of RichFunction to reduce the possibility of blocking processing
records.
Best
Yun Tang
From: Alexey Trenikhun
Sent: Tuesday, January 28, 2020 15:15
To: Yun
https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#kafka-consumers-offset-committing-behaviour-configuration
Best
Yun Tang
From: Ahmad Hassan
Sent: Tuesday, January 28, 2020 17:43
To: user
Subject: Re: Flink RocksDB logs filling up disk
https://github.com/apache/flink/blob/7e1a0f446e018681cb537dd936ae54388b5a7523/flink-core/src/main/java/org/apache/flink/configuration/TaskManagerOptions.java#L158
Best
Yun Tang
From: Li Peng
Sent: Thursday, January 30, 2020 9:24
To: user
Subject: Task-manager kuber
nk/runtime/jobmaster/JobMaster.java
[3]
https://github.com/apache/flink/blob/release-1.7/flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
[4] https://issues.apache.org/jira/browse/FLINK-4319
Best
Yun Tang
From: Lu Weizheng
Se
Great work, thanks Gary & Yu !
Best
Yun Tang
From: Wyatt Chun
Sent: Wednesday, February 12, 2020 21:36
To: Yu Li
Cc: user
Subject: Re: [ANNOUNCE] Apache Flink 1.10.0 released
Sounds great. Congrats & Thanks!
On Wed, Feb 12, 2020 at 9:31 PM Yu Li
ma
Hi
Yes, you are right. Just simply use checkpoint subtask_id -1 would find the
corresponding task subtask_id.
Best
Yun Tang
From: 杨东晓
Sent: Friday, February 14, 2020 10:11
To: user
Subject: The mapping relationship between Checkpoint subtask id and Task
should not come from Flink if you're using Flink-1.5+
[1] https://issues.apache.org/jira/browse/FLINK-8540
Best
Yun Tang
From: Piyush Narang
Sent: Saturday, March 7, 2020 6:15
To: user
Subject: Understanding n LIST calls as part of checkpointing
Hi fo
compaction_filter.h#L140
Best
Yun Tang
From: LakeShen
Sent: Tuesday, March 17, 2020 15:30
To: dev ; user-zh ; user
Subject: Question about RocksDBStateBackend Compaction Filter state cleanup
Hi community ,
I see the flink RocksDBStateBackend state
Hi Fanbin
To resume from checkpoint, you should provide at least the directory named as
/path/chk-x or /path/chk-x/_metadata. The sub-dir named as “shared” is used to
store incremental checkpoint content. You could refer to [1] for more
information.
BTW, stop with savepoint could help reduce
Very interesting and convenient tool, just a quick question: could this tool
also handle deployment cluster commands like "-tm" mixed with configuration in
`flink-conf.yaml` ?
Best
Yun Tang
From: Yangze Guo
Sent: Friday, March 27, 2020 18:00
To: u
path, I am
afraid this cannot understand hdfs paths.
[1]
https://github.com/apache/flink/blob/ae3b0ff80b93a83a358ab474060473863d2c30d6/flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/BootstrapTools.java#L420
Best
Yun Tang
From: Vitaliy Sem
Hi Lasse
Never meet this problem before, but can you share some exception stack trace so
that we could take a look. The simple project to reproduce is also a good
choice.
Best
Yun Tang
From: Lasse Nedergaard
Sent: Tuesday, March 31, 2020 19:10
To: user
/src/main/java/org/apache/flink/runtime/checkpoint/Checkpoints.java#L96
Best
Yun Tang
From: Shachar Carmeli
Sent: Tuesday, April 7, 2020 16:19
To: user@flink.apache.org
Subject: Flink incremental checkpointing - how long does data is kept in the
share folder
We
/flink/streaming/connectors/kafka/FlinkKafkaProducer.java#L871
Best
Yun Tang
From: Maxim Parkachov
Sent: Monday, April 6, 2020 23:16
To: user@flink.apache.org
Subject: New kafka producer on each checkpoint
Hi everyone,
I'm trying to test exactly once function
quot;models" is just a HashMap[(String, String), Model], and I don't know why we
need to couple all models to just one specific key.
Best
Yun Tang
From: Salva Alcántara
Sent: Sunday, April 5, 2020 20:22
To: user@flink.apache.org
Subject: Re: Using MapSt
Excited to see the stateful functions release!
Thanks for the great work of manager Gordon and everyone who ever contributed
to this.
Best
Yun Tang
From: Till Rohrmann
Sent: Wednesday, April 8, 2020 14:30
To: dev
Cc: Oytun Tez ; user
Subject: Re: [ANNOUNCE
/ClusterEntrypoint.java#L260
[2]
https://github.com/apache/flink/blob/d7e247209358779b6485062b69965b83043fb59d/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/ZooKeeperCompletedCheckpointStore.java#L234
Best
Yun Tang
From: Marc LEGER
Sent: Wednesday
heckpoints would occupy about 585MB memory, which is close to your
observed scenario.
>From my point of view, the checkpoint interval of one second is really too
>often and would not make much sense in production environment.
Best
Yun Tang
From: Till Rohrmann
org/apache/flink/contrib/streaming/state/RocksDBListState.java#L151
Best
Yun Tang
From: anaray
Sent: Friday, April 10, 2020 1:25
To: user@flink.apache.org
Subject: FlinkRuntimeException: Unexpected list element deserialization failure
Hi flink team,
I see below
ointMetadata to load '_metadata' to know which files
belonging to that checkpoint.
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/checkpoints.html#directory-structure
Best
Yun Tang
From: Shachar Carmeli
Sent: Sunday,
async time would not be too large, the most common
reason is operator receiving the barrier late which lead to the end-to-end
duration large. I hope you could offer the UI of your checkpoint details for
further investigation.
[1] https://issues.apache.org/jira/browse/FLINK-13390
Best
Yun Tang
could be run well as expected.
Best
Yun Tang
From: Till Rohrmann
Sent: Wednesday, April 15, 2020 16:08
To: dev
Cc: Eduardo Winpenny Tejedor ; Seth Wiesman
; Niels Basjes ; user
Subject: Re: [PROPOSAL] Contribute training materials to Apache Flink
Hi David
().getBroadcastState ? Did you pass a different
operator state descriptor each time?
Best
Yun Tang
From: Stephen Patel
Sent: Thursday, April 16, 2020 2:09
To: user@flink.apache.org
Subject: Streaming Job eventually begins failing during checkpointing
I've got a flink (
/4fc924a8193bb9495c6b7ba755ced576bb8a35d5/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/stableinput/BufferingDoFnRunner.java#L95
Best
Yun Tang
From: Stephen Patel
Sent: Thursday, April 16, 2020 22:30
To: Yun Tang
Cc: user@flink.apache.org
Subject
Hi Kent
You can view checkpoint details via web UI to know how much checkpointed data
uploaded for each operator, and you can compare the state size as time goes on
to see whether they upload checkpointed data in stable range.
Best
Yun Tang
From: Kent Murra
101 - 200 of 341 matches
Mail list logo
