Bom Dia Vinicius,
Can You still find (and post) the exception stack from your jobmanager log, the
flink client log does not reveal enough information.
Your situation reminds me of something similar I had.
In the log you might find something like this or similar:
2022-03-07 02:15:41,347 INFO
Hello everyone,
First of all, sorry for cross posting, I asked on SO, but David Anderson
suggested me to reach out to the community via the mailing list. The link
to the SO question is the following:
Hi Alexey,
To my best knowledge it's lazy with RocksDBStateBackend, using the Java
iterator you could even modify the map (e.g. remove()).
Cheers
Thias
From: Alexey Trenikhun
Sent: Dienstag, 8. März 2022 06:11
To: Flink User Mail List
Subject: MapState.entries()
Hello,
We are using
*Hello Flink Team,*
I am right now using Flink stateful function in my project, which are
consuming avro serialized events(both key and value are serialized) from
kafka, but it seems there is no configuration that users can customize for
deserializing the kafka record's key, because I noticed
Hello Team,
What happens after checkpoint timeout?
Does Flink reprocess data from the previous checkpoint for all tasks?
I have one compute intensive operator with parallelism of 20 and only one
of the parallel tasks seems to get stuck because of data skew. On
checkpoint timeout , will data be
Hi Diwakar,
An asynchronous flatmap function without the support of the framework can
be problematic. You should not call collector.collect outside the main
thread of the task, i.e. outside the flatMap method.
I'd suggest using a customized Source instead to process the files, which
uses a
图挂了
https://postimg.cc/Z9XdxwSk
在 2022-03-08 14:05:39，"Sun.Zhu" <17626017...@163.com> 写道：
hi all，
flink 1.13.2，将checkpoint 写到S3但是一直成功不了，一直显示IN_PROGRESS，直到超时失败，有大佬遇到过吗？
hi all，
flink 1.13.2，将checkpoint 写到S3但是一直成功不了，一直显示IN_PROGRESS，直到超时失败，有大佬遇到过吗？
Hello,
We are using RocksDBStateBackend, is MapState.entries() call in this case
"lazy" - deserializes single entry while next(), or MapState.entries() returns
collection, which is fully loaded into memory?
Thanks,
Alexey
Hi De Xun,
Unfortunately MAP, ARRAY and ROW types are supported by Flink Parquet format
only since Flink 1.15 (see FLINK-17782 [1], not released yet). You may want to
upgrade Flink version to 1.15 once it is released, or make your own
implementation based on the latest code on master branch
Hi Dawid,
My team's Flink application's primary purpose is not to count the number of SQS
messages received or the number of successful or failed S3 downloads. The
application's primary purpose is to process events and the corresponding data,
and for each event, create or update a new entry in
Hi Yun,
Thank you for the response.
1. You could tune your job to avoid backpressure. Maybe you can upgrade
your flink engine to at least flink-1.13 to know how to monitor the back
pressure status [1].
[VIDYA] - In the view of my organization, it's a very big activity to
upgrade to
HI ,
谢谢，有没有好的解决方案解决该问题呀？
在 2022-03-08 02:20:57，"Zhilong Hong" 写道：
>Hi, 明文：
>
>这个报错实际上是TM失联，一般是TM被kill导致的，可以根据TM的Flink日志和GC日志、集群层面的NM日志（YARN环境）或者是K8S日志查看TM被kill的原因。一般情况下可能是：gc时间过长导致TM心跳超时被kill、TM内存超用导致container/pod被kill等等。
>
>Best.
>Zhilong
>
>On Mon, Mar 7, 2022 at 10:18 AM 潘明文
Hello Everyone,
I'm running a streaming application using Flink 1.11 and EMR 6.01. My use
case is reading files from a s3 bucket, filter file contents ( say record)
and enrich each record. Filter records and output to a sink.
I'm reading 6k files per 15mints and the total number of records is 3
Hi Dawid, thanks for the reply.
The job was still in progress and producing events. Unfortunately I was not
able to stop the job with a savepoint or to just create a savepoint. I had
to stop the job without the savepoint and restore the state using the last
checkpoint. Still reviewing my
And the comparator function
The order of the return 1,0,-1 is relevant .
In this order -1,0,1 it will sort descending I discovered.
public static class SortEventsHandlingTime implements
Comparator> {
// Let's compare 2 Tuple4 objects
public int compare(Tuple4 o1,
Tuple4 o2)
{
Hi, 明文：
这个报错实际上是TM失联，一般是TM被kill导致的，可以根据TM的Flink日志和GC日志、集群层面的NM日志（YARN环境）或者是K8S日志查看TM被kill的原因。一般情况下可能是：gc时间过长导致TM心跳超时被kill、TM内存超用导致container/pod被kill等等。
Best.
Zhilong
On Mon, Mar 7, 2022 at 10:18 AM 潘明文 wrote:
> HI 读kafka，入hbase和kafka
> flink任务经常性报错
>
>
Hi, Puneet:
Like Terry says, if you find your job failed unexpectedly, you could check
the configuration restart-strategy in your flink-conf.yaml. If the restart
strategy is set to be disabled or none, the job will transition to failed
once it encounters a failover. The job would also fail itself
Hi Terry Wang,
So adding to above provided context.. whenever task manager goes down, jobs go
into failed state and do not restart. Even though there are good enough free
slots available on other task manager to get restarted on.
Regards,
Puneet
> On 04-Mar-2022, at 4:54 PM, Terry Wang
Hi,
From the exception it seems the job has been already done when you're
triggering the savepoint.
Best,
Dawid
On 07/03/2022 14:56, Vinicius Peracini wrote:
Hello everyone,
I have a Flink job (version 1.14.0 running on EMR) and I'm having this
issue while trying to stop a job with a
Hi Shane,
I don't think counters, or should I say metrics, are the right
abstraction for the use case you described. Metrics are a way to get an
insight into the running job and what is its current state. It is not a
good mean to calculate results. Metrics are not stateful, they are not
For the record. So that other unexperienced people my benefit too
List> inputList = new ArrayList<>();
input.forEach(inputList::add);
inputList.sort(new SortEventsHandlingTime());
for (Tuple4 in: inputList){
Op ma 7 mrt. 2022 om 03:05 schreef yidan zhao :
> Collect the elements to a list,
Hello everyone,
I have a Flink job (version 1.14.0 running on EMR) and I'm having this
issue while trying to stop a job with a savepoint on S3:
org.apache.flink.util.FlinkException: Could not stop with a savepoint job
"df3a3c590fabac737a17f1160c21094c".
at
Hi,
Thanks Martijn for driving this discussion. Your concerns are very
rational.
We should do our best to keep the Flink development on the right track. I
would suggest discussing it in a vision/goal oriented way. Since Flink has
a clear vision of unified batch and stream processing, supporting
Hi Filip,
Curator 4.2 is compatible with ZK 3.4 series. When it detects that ZK 3.4
client is used, it will operate in ZK 3.4 compatible mode and won't call
org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.quorum.flexible.QuorumMaj.(Ljava/util/Map;).
Curator detects whether the
Hi Alexis,
> Is it possible to have different values of max parallelism in different
> operators?
Yes, it is possible, please refer to [1] and [2] for API details.
> I did a test in which my source had a max parallelism of 3, whereas a
> downstream operator had a (non-max) parallelism
可以尝试使用hive 2.3.7 版本，这个里面解决了一部分 hive 不兼容jdk11的问题，我们线上已经使用一年多了
At 2021-11-22 12:00:07, "aiden" <18765295...@163.com> wrote:
>求助，jdk从8升级到11后使用hive作为flink
>table的catalog抛错，排查是bsTableEnv.registerCatalog(catalogName, catalog) 抛错，具体异常为：
>11:55:22.343 [main] ERROR hive.log - Got
退订
Hi Martijn,
Thanks for driving this discussion.
+1 on efforts on more hive syntax compatibility.
With the efforts on batch processing in recent versions(1.10~1.15), many
users have run batch processing jobs based on Flink.
In our team, we are trying to migrate most of the existing online batch
k8s native session 模式下，配置了ha，job_manager 的数量为3，然后web ui，通过rest
service访问，总是路由到非master节点，有什么办法使其稳定吗？
Hi everyone,
Flink currently has 4 APIs with multiple language support which can be used
to develop applications:
* DataStream API, both Java and Scala
* Table API, both Java and Scala
* Flink SQL, both in Flink query syntax and Hive query syntax (partially)
* Python API
Since FLIP-152 [1] the
退订
??insert into x() values() ON duplicate
KeyInsert update
----
??: "payne_z"
Hi Zhanghao
it's 3.5.5
Thank you
Fil
On Sat, 5 Mar 2022 at 08:12, Zhanghao Chen
wrote:
> Hi Filip,
>
> Could you share the version of the ZK server you are connecting to?
>
>
> Best,
> Zhanghao Chen
> --
> *From:* Filip Karnicki
> *Sent:* Friday, March 4, 2022
34 matches
Mail list logo
