I am seeing an issue with class loaders not being GCed and the metaspace
eventually OOM. Here is my setup:
- Flink 1.13.1 on EMR using JDK 8 in session mode
- Job manager is a long-running yarn session
- New jobs are submitted every 5m (and typically run for less than 5m)
I find that after a few
far as I remember this is a known issue a few years ago but Flink
>> currently has no solution to this (correct me if I'm wrong). I see that
>> you're running jobs on a yarn session. Could you switch to yarn-per-job
>> mode (where JM and TMs are created and destroyed for ea
I'm seeing files orphaned in HDFS and wondering how to clean them up when
the job is completed. The directory is /user/yarn/.flink so I am assuming
this is created by flink? The HDFS in my cluster eventually fills up.
Here is my setup:
- Flink 1.13.1 on AWS EMR
- Executing flink in per-jo
case, then having the residual staging directory is an
> expected behavior since Flink JobManager does not have a chance to do the
> clean-up.
>
>
>
> Best,
> Yang
>
> David Clutter 于2022年1月11日周二 10:08写道：
>
>> I'm seeing files orphaned in HDFS and wondering
I am using Flink 1.13.1 on AWS EMR 6.4. I have an existing application
using DataStream API that I would like to modify to write output to S3. I
am testing the StreamingFileSink with a bounded input. I have enabled
checkpointing.
A couple questions:
1) When the program finishes, all the files r
I am using Flink 1.13.1 on AWS EMR and I seem to have hit this bug:
https://issues.apache.org/jira/browse/FLINK-27041. My job will fail when
there are empty partitions. I see it is fixed in a newer version of Flink
but I cannot update Flink version at this time.
Suggestions on a workaround? I a
