Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

[Jiadong Lu]

Congratulations !!!

Best,
Jiadong Lu

On 2023/3/27 17:23, Yu Li wrote:

Dear Flinkers,


As you may have noticed, we are pleased to announce that Flink Table Store has 
joined the Apache Incubator as a separate project called Apache 
Paimon(incubating) [1] [2] [3]. The new project still aims at building a 
streaming data lake platform for high-speed data ingestion, change data 
tracking and efficient real-time analytics, with the vision of supporting a 
larger ecosystem and establishing a vibrant and neutral open source community.


We would like to thank everyone for their great support and efforts for the 
Flink Table Store project, and warmly welcome everyone to join the development 
and activities of the new project. Apache Flink will continue to be one of the 
first-class citizens supported by Paimon, and we believe that the Flink and 
Paimon communities will maintain close cooperation.


亲爱的Flinkers，


正如您可能已经注意到的，我们很高兴地宣布，Flink Table Store 已经正式加入 
Apache 孵化器独立孵化 [1] [2] [3]。新项目的名字是 
Apache Paimon(incubating)，仍致力于打造一个支持高速数据摄入、流式数据订 
阅和高效实时分析的新一代流式湖仓平台。此外，新项目将支持更加丰富的生态， 
并建立一个充满活力和中立的开源社区。



在这里我们要感谢大家对 Flink Table Store 项目的大力支持和投入，并热烈欢 
迎大家加入新项目的开发和社区活动。Apache Flink 将继续作为 Paimon 支持的 
主力计算引擎之一，我们也相信 Flink 和 Paimon 社区将继续保持密切合作。



Best Regards,

Yu (on behalf of the Apache Flink PMC and Apache Paimon PPMC)


致礼，

李钰（谨代表 Apache Flink PMC 和 Apache Paimon PPMC）


[1] https://paimon.apache.org/ 

[2] https://github.com/apache/incubator-paimon 



[3] https://cwiki.apache.org/confluence/display/INCUBATOR/PaimonProposal 

Re: Is MiniClusterWithClientResource expected to have a UI?

[Weihua Hu]

Hi, Sofya

The MiniClusterWithClientResource does not provide UI by default.
But you can enable it by adding flink-runtime-web dependency to do some
debug.


Add this dependency to your pom.xml. And flink will load the web ui
automatically.


   org.apache.flink
   flink-runtime-web
   ${project.version}
   test



Best,
Weihua


On Thu, Mar 30, 2023 at 5:47 AM Sofya T. Irwin  wrote:

> Hi,
>
> I'm experimenting with the MiniClusterWithClientResource, below, and when
> I print out the URL I'm not able to access a UI. Is
> the MiniClusterWithClientResource expected to provide a UI?
>
> Thanks
> -Sofya
>
>
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/datastream/testing/#junit-rule-miniclusterwithclientresource
>
>
> public static MiniClusterWithClientResource flinkCluster =
> new MiniClusterWithClientResource(
>  new MiniClusterResourceConfiguration.Builder()
>  .setNumberSlotsPerTaskManager(2)
>  .setNumberTaskManagers(1)
>  .build());
>
> ClusterClient clusterClient = flinkCluster.getClusterClient();
> RestClusterClient restClusterClient =
> flinkCluster.getRestClusterClient();
> String URL = clusterClient.getWebInterfaceURL();
> String URL2 = restClusterClient.getWebInterfaceURL();
> System.out.println("URL " + URL); // Provided URL is not accessible
>
>
>
>
>

Docker Flink Official Image with Docker Swarm cannot register TaskManager to JobManager's Resource Manager

[chyp long]

Hello, I am having trouble when deploying Apache Flink 1.16.1 on 2 Google
Cloud instances with Docker Swarm. The JobManager is deployed on the
manager node and the TaskManager is deployed in the worker node. The
TaskManager seems to have trouble to communicate with ResourceManager on
JobManager via akka.tcp. Here is the log on the TaskManager

flink_taskmanager.1.e7sxy43ls...@workernode.gcp| 2023-03-29
13:50:34,061 INFO
org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService [] -
Start job leader service.
flink_taskmanager.1.e7sxy43ls...@workernode.gcp| 2023-03-29
13:50:34,066 INFO  org.apache.flink.runtime.filecache.FileCache
 [] - User file cache uses directory
/tmp/flink-dist-cache-705fdca5-c285-4875-9b68-556ccd1b56c3
flink_taskmanager.1.e7sxy43ls...@workernode.gcp| 2023-03-29
13:50:34,073 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor
 [] - Connecting to ResourceManager
akka.tcp://flink@jobmanager:6123/user/rpc/resourcemanager_*().
flink_taskmanager.1.e7sxy43ls...@workernode.gcp| 2023-03-29
13:50:34,420 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor
 [] - Resolved ResourceManager address, beginning registration
flink_taskmanager.1.e7sxy43ls...@workernode.gcp| 2023-03-29
13:55:34,086 ERROR org.apache.flink.runtime.taskexecutor.TaskExecutor
 [] - Fatal error occurred in TaskExecutor
akka.tcp://flink@10.0.1.14:6127/user/rpc/taskmanager_0.
flink_taskmanager.1.e7sxy43ls...@workernode.gcp|
org.apache.flink.runtime.taskexecutor.exceptions.RegistrationTimeoutException:
Could not register at the ResourceManager within the specified maximum
registration duration PT5M. This indicates a problem with this
instance. Terminating now.

About firewall rules, these instances are deployed in the same VPC,
same subnet, so I suppose it can communicate without any troubles. In
fact, I can ping or curl on both instances. Before deploying to Google
Cloud, I have successfully deployed the same setup on my private
Microstack cloud without any problem. Here is my docker-compose file
that I used docker stack deploy:

version: '3.8'

services:
  jobmanager:
image: halo93/fixed-ports-flink-docker:1.16.1-scala_2.12-java11-custom
deploy:
  replicas: 1
  placement:
constraints: [node.hostname == managernode.gcp]
ports:
  - "8081:8081"
  - "6123:6123"
  - "6124:6124"
  - "6125:6125"
command: jobmanager
environment:
  - FLINK_PROPERTIES=${FLINK_PROPERTIES}
networks:
  - flink-network
  taskmanager:
image: halo93/fixed-ports-flink-docker:1.16.1-scala_2.12-java11-custom
deploy:
  replicas: 1
  placement:
constraints: [node.hostname == workernode.gcp]
depends_on:
  - jobmanager
ports:
  - "6121:6121"
  - "6122:6122"
  - "6126:6126"
  - "6127:6127"
  - "6128:6128"
  - "5005:5005/udp"
command:
  - taskmanager
environment:
  - FLINK_PROPERTIES=${FLINK_PROPERTIES}
networks:
  - flink-network

networks:
  flink-network:
driver: overlay
attachable: true

FLINK_PROPERTIES

FLINK_PROPERTIES=$'\njobmanager.rpc.address:
jobmanager\nparallelism.default: 2\n'

I am using a customized flink docker image to fix taskmanager.data.port and
taskmanager.rpc.port to 6126 and 6127. I have tried to change
jobmanager.rpc.address with the private IP and zonal DNS, ResourceManager
can register the taskmanager. However, by doing so, flink-metrics is unable
to work. I expect that I can successfully deploy flink cluster on Google
Cloud instances with Docker Swarm, same with what I did on Microstack.

Is MiniClusterWithClientResource expected to have a UI?

[Sofya T. Irwin]

Hi,

I'm experimenting with the MiniClusterWithClientResource, below, and when I
print out the URL I'm not able to access a UI. Is
the MiniClusterWithClientResource expected to provide a UI?

Thanks
-Sofya

https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/datastream/testing/#junit-rule-miniclusterwithclientresource


public static MiniClusterWithClientResource flinkCluster =
new MiniClusterWithClientResource(
 new MiniClusterResourceConfiguration.Builder()
 .setNumberSlotsPerTaskManager(2)
 .setNumberTaskManagers(1)
 .build());

ClusterClient clusterClient = flinkCluster.getClusterClient();
RestClusterClient restClusterClient =
flinkCluster.getRestClusterClient();
String URL = clusterClient.getWebInterfaceURL();
String URL2 = restClusterClient.getWebInterfaceURL();
System.out.println("URL " + URL); // Provided URL is not accessible

RE: [EXTERNAL] Re: Unable to Use spec.flinkVersion v1_17 with Flink Operator

[Ivan Webber via user]

The commands for updating CRD seem to assume they are being run from the 
flink-operator-repo. Accordingly, to run them in my environment I should run 
them as:
```bash
# upgrade CRD
kubectl replace -f 
https://github.com/apache/flink-kubernetes-operator/tree/release-1.4/helm/flink-kubernetes-operator/crds/flinkdeployments.flink.apache.org-v1.yml
kubectl replace -f 
https://github.com/apache/flink-kubernetes-operator/tree/release-1.4/helm/flink-kubernetes-operator/crds/flinksessionjobs.flink.apache.org-v1.yml
```

However, I got the following error for both commands:
```bash
error: error parsing 
https://github.com/apache/flink-kubernetes-operator/tree/release-1.4/helm/flink-kubernetes-operator/crds/flinkdeployments.flink.apache.org-v1.yml:
 error converting YAML to JSON: yaml: line 27: mapping values are not allowed 
in this context
```

I tried the following based on the documentation:
```bash

helm uninstall flink-kubernetes-operator

kubectl delete crd flinkdeployments.flink.apache.org

helm repo update flink-operator-repo
helm install flink-kubernetes-operator 
flink-operator-repo/flink-kubernetes-operator
```

Now I do not get the error about `v1_17` and my pipeline is starting. I will 
follow-up if I find any new problems.

Thanks for the help!

Best,

Ivan


From: Gyula Fóra
Sent: Tuesday, March 28, 2023 10:25 PM
To: Ivan Webber
Cc: Ivan Webber via user
Subject: [EXTERNAL] Re: Unable to Use spec.flinkVersion v1_17 with Flink 
Operator

You don't often get email from gyula.f...@gmail.com. Learn why this is 
important
I think you forgot to upgrade the CRD during the upgrade process on your 
cluster.

As you can see here: 
https://github.com/apache/flink-kubernetes-operator/blob/release-1.4/helm/flink-kubernetes-operator/crds/flinkdeployments.flink.apache.org-v1.yml#L38-L44
The newer version already contains support for 1.17.
For docs you can refer to: 
https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/upgrade/#1-upgrading-the-crd


Cheers,
Gyula

On Tue, Mar 28, 2023 at 10:29 PM Ivan Webber via user 
mailto:user@flink.apache.org>> wrote:
Hello Flink Users:

I'm trying to upgrade to use Flink 1.17.0 with my pipeline in order to have 
support for writing to Azure Data Lake Storage. However when I change the 
`spec.flinkVersion` to v1_17 I get an error message:

```bash
The FlinkDeployment "test-replay-streaming-run" is invalid: spec.flinkVersion: 
Unsupported value: "v1_17": supported values: "v1_13", "v1_14", "v1_15", "v1_16"
```

The documentation 
(here)
 says latest version of Flink Operator (1.4) should support 1.17.0 and I made 
sure to update by running the below commands. I’m wondering if an Enum needs 
updated or if the latest stable Flink Operator doesn’t actually support 1.17.0 
yet. Any pointers would be appreciated.

```bash
helm uninstall flink-kubernetes-operator 
flink-operator-repo/flink-kubernetes-operator
helm repo rm flink-operator-repo
helm repo add flink-operator-repo 
https://downloads.apache.org/flink/flink-kubernetes-operator-1.4.0/
helm install flink-kubernetes-operator 
flink-operator-repo/flink-kubernetes-operator
```

Thanks,

Ivan Webber

Re: Unexpected behaviour when configuring both `withWatermarkAlignment` & `withIdleness`

[Reem Razak via user]

Hey Martijn,

The version is 1.16.0

On Wed, Mar 29, 2023 at 5:43 PM Martijn Visser 
wrote:

> Hi Reem,
>
> What's the Flink version where you're encountering this issue?
>
> Best regards,
>
> Martijn
>
> On Wed, Mar 29, 2023 at 5:18 PM Reem Razak via user 
> wrote:
>
>> Hey there!
>>
>> We are seeing a second Flink pipeline encountering similar issues when
>> configuring both `withWatermarkAlignment` and `withIdleness`. The
>> unexpected behaviour gets triggered after a Kafka cluster failover. Any
>> thoughts on there being an incompatibility between the two?
>>
>> Thanks!
>>
>> On Wed, Nov 9, 2022 at 6:42 PM Reem Razak  wrote:
>>
>>> Hi there,
>>>
>>> We are integrating the watermark alignment feature into a pipeline with
>>> a Kafka source during a "backfill"- i.e. playing from an earlier Kafka
>>> offset. While testing, we noticed some unexpected behaviour in the
>>> watermark advancement which was resolved by removing `withIdleness` from
>>> our watermark strategy.
>>>
>>>
>>> val watermarkStrategy = WatermarkStrategy
>>>   .forBoundedOutOfOrderness(Duration.ofMinutes(1))
>>>   .withTimestampAssigner(new
>>> TimestampedEventTimestampAssigner[Event])
>>>   .withWatermarkAlignment("alignment-group-1", Duration.ofMinutes(1))
>>>   .withIdleness(Duration.ofMinutes(5))
>>>
>>> I have attached a couple of screenshots of the watermarkAlignmentDrift
>>> metric. As you can see, the behaviour seems normal until a sudden drop in
>>> the value to ~ Long.MIN_VALUE, which causes the pipeline to stop emitting
>>> records completely from the source. Furthermore, the logs originating from
>>> from
>>> https://github.com/dawidwys/flink/blob/888162083fbb5f62f809e5270ad968db58cc9c5b/flink-runtime/src/main/java/org/apache/flink/runtime/source/coordinator/SourceCoordinator.java#L174
>>> also indicate that the `maxAllowedWatermark` switches to ~ Long.MIN_VALUE.
>>>
>>> We found that modifying the `updateInterval` passed into the alignment
>>> parameters seemed to correlate with how long the pipeline would operate
>>> before stopping - a larger interval of 20 minutes would encounter the issue
>>> later than an interval of 1 second.
>>>
>>> We are wondering if a bug exists when using both `withIdleness` and
>>> `withWatermarkAlignment`. Might it be related to
>>> https://issues.apache.org/jira/browse/FLINK-28975, or is there possibly
>>> a race condition in the watermark emission? We do not necessarily need to
>>> have both configured at the same time, but we were also surprised by the
>>> behaviour of the application. Has anyone run into a similar issue or have
>>> further insight?
>>>
>>> Much Appreciated,
>>> - Reem
>>>
>>>
>>>
>>>

Re: Unexpected behaviour when configuring both `withWatermarkAlignment` & `withIdleness`

[Martijn Visser]

Hi Reem,

What's the Flink version where you're encountering this issue?

Best regards,

Martijn

On Wed, Mar 29, 2023 at 5:18 PM Reem Razak via user 
wrote:

> Hey there!
>
> We are seeing a second Flink pipeline encountering similar issues when
> configuring both `withWatermarkAlignment` and `withIdleness`. The
> unexpected behaviour gets triggered after a Kafka cluster failover. Any
> thoughts on there being an incompatibility between the two?
>
> Thanks!
>
> On Wed, Nov 9, 2022 at 6:42 PM Reem Razak  wrote:
>
>> Hi there,
>>
>> We are integrating the watermark alignment feature into a pipeline with a
>> Kafka source during a "backfill"- i.e. playing from an earlier Kafka
>> offset. While testing, we noticed some unexpected behaviour in the
>> watermark advancement which was resolved by removing `withIdleness` from
>> our watermark strategy.
>>
>>
>> val watermarkStrategy = WatermarkStrategy
>>   .forBoundedOutOfOrderness(Duration.ofMinutes(1))
>>   .withTimestampAssigner(new TimestampedEventTimestampAssigner[Event])
>>   .withWatermarkAlignment("alignment-group-1", Duration.ofMinutes(1))
>>   .withIdleness(Duration.ofMinutes(5))
>>
>> I have attached a couple of screenshots of the watermarkAlignmentDrift
>> metric. As you can see, the behaviour seems normal until a sudden drop in
>> the value to ~ Long.MIN_VALUE, which causes the pipeline to stop emitting
>> records completely from the source. Furthermore, the logs originating from
>> from
>> https://github.com/dawidwys/flink/blob/888162083fbb5f62f809e5270ad968db58cc9c5b/flink-runtime/src/main/java/org/apache/flink/runtime/source/coordinator/SourceCoordinator.java#L174
>> also indicate that the `maxAllowedWatermark` switches to ~ Long.MIN_VALUE.
>>
>> We found that modifying the `updateInterval` passed into the alignment
>> parameters seemed to correlate with how long the pipeline would operate
>> before stopping - a larger interval of 20 minutes would encounter the issue
>> later than an interval of 1 second.
>>
>> We are wondering if a bug exists when using both `withIdleness` and
>> `withWatermarkAlignment`. Might it be related to
>> https://issues.apache.org/jira/browse/FLINK-28975, or is there possibly
>> a race condition in the watermark emission? We do not necessarily need to
>> have both configured at the same time, but we were also surprised by the
>> behaviour of the application. Has anyone run into a similar issue or have
>> further insight?
>>
>> Much Appreciated,
>> - Reem
>>
>>
>>
>>

Re: Unexpected behaviour when configuring both `withWatermarkAlignment` & `withIdleness`

[Reem Razak via user]

Hey there!

We are seeing a second Flink pipeline encountering similar issues when
configuring both `withWatermarkAlignment` and `withIdleness`. The
unexpected behaviour gets triggered after a Kafka cluster failover. Any
thoughts on there being an incompatibility between the two?

Thanks!

On Wed, Nov 9, 2022 at 6:42 PM Reem Razak  wrote:

> Hi there,
>
> We are integrating the watermark alignment feature into a pipeline with a
> Kafka source during a "backfill"- i.e. playing from an earlier Kafka
> offset. While testing, we noticed some unexpected behaviour in the
> watermark advancement which was resolved by removing `withIdleness` from
> our watermark strategy.
>
>
> val watermarkStrategy = WatermarkStrategy
>   .forBoundedOutOfOrderness(Duration.ofMinutes(1))
>   .withTimestampAssigner(new TimestampedEventTimestampAssigner[Event])
>   .withWatermarkAlignment("alignment-group-1", Duration.ofMinutes(1))
>   .withIdleness(Duration.ofMinutes(5))
>
> I have attached a couple of screenshots of the watermarkAlignmentDrift
> metric. As you can see, the behaviour seems normal until a sudden drop in
> the value to ~ Long.MIN_VALUE, which causes the pipeline to stop emitting
> records completely from the source. Furthermore, the logs originating from
> from
> https://github.com/dawidwys/flink/blob/888162083fbb5f62f809e5270ad968db58cc9c5b/flink-runtime/src/main/java/org/apache/flink/runtime/source/coordinator/SourceCoordinator.java#L174
> also indicate that the `maxAllowedWatermark` switches to ~ Long.MIN_VALUE.
>
> We found that modifying the `updateInterval` passed into the alignment
> parameters seemed to correlate with how long the pipeline would operate
> before stopping - a larger interval of 20 minutes would encounter the issue
> later than an interval of 1 second.
>
> We are wondering if a bug exists when using both `withIdleness` and
> `withWatermarkAlignment`. Might it be related to
> https://issues.apache.org/jira/browse/FLINK-28975, or is there possibly a
> race condition in the watermark emission? We do not necessarily need to
> have both configured at the same time, but we were also surprised by the
> behaviour of the application. Has anyone run into a similar issue or have
> further insight?
>
> Much Appreciated,
> - Reem
>
>
>
>

Re: flink sql的codegen导致metaspace OOM疑问

[Shammon FY]

Hi

自增id可以为同一个作业的多个codegen类生成唯一类名
一般metaspace可以通过fullgc释放，你可以查看你的集群metaspace大小，是否触发了了fullgc

Best,
Shammon FY

On Wednesday, March 29, 2023, tanjialiang  wrote:

> Hi all,
>我有一个通过flink kubernetes operator定时提交到同一个session作业（底层是将flink
> sql转JobGraph的逻辑下推到了JobManager执行），当他跑了一段时间后，JobManager报了metaspace OOM.
>经过排查后发现是flink sql codegen生成的代码类有一个自增ID，这些类在使用完后不会释放。
>
>
> 疑问：
> 1. flink sql codegen做这样的一个自增ID有什么特殊意义吗？
> 2. java中通过类加载器加载的类有什么办法可以释放？
>
>
>
>
>

Re: org.apache.flink.table.api.ValidationException

[Hang Ruan]

Hi,

This error occurs when the data type can not be parsed. You could read this
part to see more details about the User-Defined Data Types[1].

Best,
Hang

[1]
https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/types/#user-defined-data-types


柒朵 <1303809...@qq.com> 于2023年3月29日周三 17:52写道：

>
> Exception in thread "main" org.apache.flink.table.api.ValidationException: 
> Could not extract a data type from 'class
> UserStatus'. Please pass the required data type manually or allow RAW types.
> --
> 柒朵
> 1303809...@qq.com
>
> 
>
>

org.apache.flink.table.api.ValidationException

[????]

Exceptioninthread"main"org.apache.flink.table.api.ValidationException:Couldnotextractadatatypefrom'class
 
UserStatus'.PleasepasstherequireddatatypemanuallyorallowRAWtypes.



1303809...@qq.com

Re: Questions on S3 File Sink Behavior

[Mate Czagany]

Hi,

1. In case of S3 FileSystem, Flink uses the multipart upload process [1]
for better performance. It might not be obvious at first by looking at the
docs, but it's noted at the bottom of the FileSystem page [2]
For more information you can also check FLINK-9751 and FLINK-9752

2. In case of local FileSystem it always starts with a dot according to
LocalRecoverableWriter [3] but make sure to check the implementation of
RecoverableWriter for the FileSystem you want to use.

Regards,
Mate

[1] https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html
[2]
https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/filesystem/#s3-specific
[3]
https://github.com/apache/flink/blob/1e0b58aa8d962469fa9dd7b470037aeaece43500/flink-core/src/main/java/org/apache/flink/core/fs/local/LocalRecoverableWriter.java#L129

Chirag Dewan via user  ezt írta (időpont: 2023.
márc. 29., Sze, 9:07):

> Hi,
>
>
>
> We are tying to use Flink's File sink to distribute files to AWS S3
> storage. We are using Flink provided Hadoop s3a connector as plugin.
>
> We have some observations that we needed to clarify:
>
> 1. When using file sink for local filesystem distribution, we can see that
> the sink creates 3 sets of files - in progress, pending (on rolling) and
> finished (upon checkpointing). But with S3 file sink we can see only the
> finished files, in the S3 buckets.
>
> So we wanted to understand where does the sink creates the in-progress and
> pending files for S3 file sink ?
>
>
> 2. We can also see with local file system sink, the in-progress and
> pending file names follow the nomenclature:
> .--.inprogress.uid-
>
> There is a dot at the begining of the filename, may be flink is trying to
> create these files as hidden files. But in the flink documentation this is
> not mentioned.
>
> So can we assume that the in-progress and pending filenames shall always
> start with a dot ?
>
> thanks a lot in advance
>
>
>

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

[Dong Lin]

Congratulations!

Dong

On Mon, Mar 27, 2023 at 5:24 PM Yu Li  wrote:

> Dear Flinkers,
>
>
>
> As you may have noticed, we are pleased to announce that Flink Table Store 
> has joined the Apache Incubator as a separate project called Apache 
> Paimon(incubating) [1] [2] [3]. The new project still aims at building a 
> streaming data lake platform for high-speed data ingestion, change data 
> tracking and efficient real-time analytics, with the vision of supporting a 
> larger ecosystem and establishing a vibrant and neutral open source community.
>
>
>
> We would like to thank everyone for their great support and efforts for the 
> Flink Table Store project, and warmly welcome everyone to join the 
> development and activities of the new project. Apache Flink will continue to 
> be one of the first-class citizens supported by Paimon, and we believe that 
> the Flink and Paimon communities will maintain close cooperation.
>
>
> 亲爱的Flinkers，
>
>
> 正如您可能已经注意到的，我们很高兴地宣布，Flink Table Store 已经正式加入 Apache
> 孵化器独立孵化 [1] [2] [3]。新项目的名字是
> Apache 
> Paimon(incubating)，仍致力于打造一个支持高速数据摄入、流式数据订阅和高效实时分析的新一代流式湖仓平台。此外，新项目将支持更加丰富的生态，并建立一个充满活力和中立的开源社区。
>
>
> 在这里我们要感谢大家对 Flink Table Store 项目的大力支持和投入，并热烈欢迎大家加入新项目的开发和社区活动。Apache Flink
> 将继续作为 Paimon 支持的主力计算引擎之一，我们也相信 Flink 和 Paimon 社区将继续保持密切合作。
>
>
> Best Regards,
>
> Yu (on behalf of the Apache Flink PMC and Apache Paimon PPMC)
>
>
> 致礼，
>
> 李钰（谨代表 Apache Flink PMC 和 Apache Paimon PPMC）
>
>
> [1] https://paimon.apache.org/
>
> [2] https://github.com/apache/incubator-paimon
>
> [3] https://cwiki.apache.org/confluence/display/INCUBATOR/PaimonProposal
>

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

[Dong Lin]

Congratulations!

Dong

On Mon, Mar 27, 2023 at 5:24 PM Yu Li  wrote:

> Dear Flinkers,
>
>
>
> As you may have noticed, we are pleased to announce that Flink Table Store 
> has joined the Apache Incubator as a separate project called Apache 
> Paimon(incubating) [1] [2] [3]. The new project still aims at building a 
> streaming data lake platform for high-speed data ingestion, change data 
> tracking and efficient real-time analytics, with the vision of supporting a 
> larger ecosystem and establishing a vibrant and neutral open source community.
>
>
>
> We would like to thank everyone for their great support and efforts for the 
> Flink Table Store project, and warmly welcome everyone to join the 
> development and activities of the new project. Apache Flink will continue to 
> be one of the first-class citizens supported by Paimon, and we believe that 
> the Flink and Paimon communities will maintain close cooperation.
>
>
> 亲爱的Flinkers，
>
>
> 正如您可能已经注意到的，我们很高兴地宣布，Flink Table Store 已经正式加入 Apache
> 孵化器独立孵化 [1] [2] [3]。新项目的名字是
> Apache 
> Paimon(incubating)，仍致力于打造一个支持高速数据摄入、流式数据订阅和高效实时分析的新一代流式湖仓平台。此外，新项目将支持更加丰富的生态，并建立一个充满活力和中立的开源社区。
>
>
> 在这里我们要感谢大家对 Flink Table Store 项目的大力支持和投入，并热烈欢迎大家加入新项目的开发和社区活动。Apache Flink
> 将继续作为 Paimon 支持的主力计算引擎之一，我们也相信 Flink 和 Paimon 社区将继续保持密切合作。
>
>
> Best Regards,
>
> Yu (on behalf of the Apache Flink PMC and Apache Paimon PPMC)
>
>
> 致礼，
>
> 李钰（谨代表 Apache Flink PMC 和 Apache Paimon PPMC）
>
>
> [1] https://paimon.apache.org/
>
> [2] https://github.com/apache/incubator-paimon
>
> [3] https://cwiki.apache.org/confluence/display/INCUBATOR/PaimonProposal
>

来自胡家发的邮件

[胡家发]

退订

flink sql的codegen导致metaspace OOM疑问

[tanjialiang]

Hi all,
   我有一个通过flink kubernetes operator定时提交到同一个session作业（底层是将flink 
sql转JobGraph的逻辑下推到了JobManager执行），当他跑了一段时间后，JobManager报了metaspace OOM.
   经过排查后发现是flink sql codegen生成的代码类有一个自增ID，这些类在使用完后不会释放。


疑问：
1. flink sql codegen做这样的一个自增ID有什么特殊意义吗？
2. java中通过类加载器加载的类有什么办法可以释放？

Questions on S3 File Sink Behavior

[Chirag Dewan via user]

Hi,

 


We are tying to use Flink's File sink to distribute files to AWS S3 storage. We 
are using Flink provided Hadoop s3a connector as plugin.
We have some observations that we needed to clarify:

1. When using file sink for local filesystem distribution, we can see that the 
sink creates 3 sets of files - in progress, pending (on rolling) and finished 
(upon checkpointing). But with S3 file sink we can see only the finished files, 
in the S3 buckets.
So we wanted to understand where does the sink creates the in-progress and 
pending files for S3 file sink ?

2. We can also see with local file system sink, the in-progress and pending 
file names follow the 
nomenclature:.--.inprogress.uid-

There is a dot at the begining of the filename, may be flink is trying to 
create these files as hidden files. But in the flink documentation this is not 
mentioned. 
So can we assume that the in-progress and pending filenames shall always start 
with a dot ?
thanks a lot in advance