Re: Re: [ANNOUNCE] Apache Paimon is graduated to Top Level Project
Congratulations! Best, Guowei On Mon, Apr 1, 2024 at 11:15 AM Feng Jin wrote: > Congratulations! > > Best, > Feng Jin > > On Mon, Apr 1, 2024 at 10:51 AM weijie guo > wrote: > >> Congratulations! >> >> Best regards, >> >> Weijie >> >> >> Hang Ruan 于2024年4月1日周一 09:49写道： >> >> > Congratulations! >> > >> > Best, >> > Hang >> > >> > Lincoln Lee 于2024年3月31日周日 00:10写道： >> > >> > > Congratulations! >> > > >> > > Best, >> > > Lincoln Lee >> > > >> > > >> > > Jark Wu 于2024年3月30日周六 22:13写道： >> > > >> > > > Congratulations! >> > > > >> > > > Best, >> > > > Jark >> > > > >> > > > On Fri, 29 Mar 2024 at 12:08, Yun Tang wrote: >> > > > >> > > > > Congratulations to all Paimon guys! >> > > > > >> > > > > Glad to see a Flink sub-project has been graduated to an Apache >> > > top-level >> > > > > project. >> > > > > >> > > > > Best >> > > > > Yun Tang >> > > > > >> > > > > >> > > > > From: Hangxiang Yu >> > > > > Sent: Friday, March 29, 2024 10:32 >> > > > > To: dev@flink.apache.org >> > > > > Subject: Re: Re: [ANNOUNCE] Apache Paimon is graduated to Top >> Level >> > > > Project >> > > > > >> > > > > Congratulations! >> > > > > >> > > > > On Fri, Mar 29, 2024 at 10:27 AM Benchao Li > > >> > > > wrote: >> > > > > >> > > > > > Congratulations! >> > > > > > >> > > > > > Zakelly Lan 于2024年3月29日周五 10:25写道： >> > > > > > > >> > > > > > > Congratulations! >> > > > > > > >> > > > > > > >> > > > > > > Best, >> > > > > > > Zakelly >> > > > > > > >> > > > > > > On Thu, Mar 28, 2024 at 10:13 PM Jing Ge >> > > > > > > > >> > > > > > wrote: >> > > > > > > >> > > > > > > > Congrats! >> > > > > > > > >> > > > > > > > Best regards, >> > > > > > > > Jing >> > > > > > > > >> > > > > > > > On Thu, Mar 28, 2024 at 1:27 PM Feifan Wang < >> > zoltar9...@163.com> >> > > > > > wrote: >> > > > > > > > >> > > > > > > > > Congratulations!—— >> > > > > > > > > >> > > > > > > > > Best regards, >> > > > > > > > > >> > > > > > > > > Feifan Wang >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > At 2024-03-28 20:02:43, "Yanfei Lei" > > >> > > > wrote: >> > > > > > > > > >Congratulations! >> > > > > > > > > > >> > > > > > > > > >Best, >> > > > > > > > > >Yanfei >> > > > > > > > > > >> > > > > > > > > >Zhanghao Chen 于2024年3月28日周四 >> > > > 19:59写道： >> > > > > > > > > >> >> > > > > > > > > >> Congratulations! >> > > > > > > > > >> >> > > > > > > > > >> Best, >> > > > > > > > > >> Zhanghao Chen >> > > > > > > > > >> >> > > > > > > > > >> From: Yu Li >> > > > > > > > > >> Sent: Thursday, March 28, 2024 15:55 >> > > > > > > > > >> To: d...@paimon.apache.org >> > > > > > > > > >> Cc: dev ; user < >> > u...@flink.apache.org >> > > > >> > > > > > > > > >> Subject: Re: [ANNOUNCE] Apache Paimon is graduated to >> Top >> > > > Level >> > > > > > > > Project >> > > > > > > > > >> >> > > > > > > > > >> CC the Flink user and dev mailing list. >> > > > > > > > > >> >> > > > > > > > > >> Paimon originated within the Flink community, initially >> > > known >> > > > as >> > > > > > Flink >> > > > > > > > > >> Table Store, and all our incubating mentors are >> members of >> > > the >> > > > > > Flink >> > > > > > > > > >> Project Management Committee. I am confident that the >> > bonds >> > > of >> > > > > > > > > >> enduring friendship and close collaboration will >> continue >> > to >> > > > > > unite the >> > > > > > > > > >> two communities. >> > > > > > > > > >> >> > > > > > > > > >> And congratulations all! >> > > > > > > > > >> >> > > > > > > > > >> Best Regards, >> > > > > > > > > >> Yu >> > > > > > > > > >> >> > > > > > > > > >> On Wed, 27 Mar 2024 at 20:35, Guojun Li < >> > > > > gjli.schna...@gmail.com> >> > > > > > > > > wrote: >> > > > > > > > > >> > >> > > > > > > > > >> > Congratulations! >> > > > > > > > > >> > >> > > > > > > > > >> > Best, >> > > > > > > > > >> > Guojun >> > > > > > > > > >> > >> > > > > > > > > >> > On Wed, Mar 27, 2024 at 5:24 PM wulin < >> > > ouyangwu...@163.com> >> > > > > > wrote: >> > > > > > > > > >> > >> > > > > > > > > >> > > Congratulations~ >> > > > > > > > > >> > > >> > > > > > > > > >> > > > 2024年3月27日 15:54，王刚 > > .INVALID> >> > > > 写道： >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > Congratulations~ >> > > > > > > > > >> > > > >> > > > > > > > > >> > > >> 2024年3月26日 10:25，Jingsong Li < >> > jingsongl...@gmail.com >> > > > >> > > > > 写道： >> > > > > > > > > >> > > >> >> > > > > > > > > >> > > >> Hi Paimon community, >> > > > > > > > > >> > > >> >> > > > > > > > > >> > > >> I’m glad to announce that the ASF board has >> > approved >> > > a >> > > > > > > > > resolution to >> > > > > > > > > >> > > >> graduate Paimon into a full Top Level Project. >> > Thanks >> > > > to >> > > > > > > > > everyone for >> > > > > > > > > >> > > >> your help to get to this point. >> > > > > > > > > >> > > >> >> > > > > > > > > >> > > >> I just created an issue to track the things we >> need >> > > to >> >
[jira] [Created] (FLINK-34981) FLIP-426: Grouping Remote State Access
Jinzhong Li created FLINK-34981: --- Summary: FLIP-426: Grouping Remote State Access Key: FLINK-34981 URL: https://issues.apache.org/jira/browse/FLINK-34981 Project: Flink Issue Type: New Feature Components: Runtime / State Backends Reporter: Jinzhong Li Fix For: 2.0.0 This is a sub-FLIP for the disaggregated state management and its related work, please read the [FLIP-423|https://cwiki.apache.org/confluence/x/R4p3EQ] first to know the whole story. I/O speed and latency are critical for overall data throughput, particularly in jobs that manage large states. Implementing multiple asynchronous I/O operations is a proven strategy to enhance throughput by increasing parallelism of I/O execution. However, simply expanding I/O parallelism can quickly hit a ceiling due to finite I/O bandwidth. Additionally, when it comes to remote storage access, the time taken for RPC round trips significantly outweighs the impact of I/O size on individual I/O performance. So a promising optimization is to merge adjacent I/O requests into a single operation and fetch multiple keys with one I/O call. This approach requires a pre-prepared batch of keys for the query and the identification of I/O operations that can be combined. In this FLIP, we focus on the implementation details for batching state requests and processing them in batches. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34980) Translate overview document into Chinese
Caican Cai created FLINK-34980: -- Summary: Translate overview document into Chinese Key: FLINK-34980 URL: https://issues.apache.org/jira/browse/FLINK-34980 Project: Flink Issue Type: Sub-task Reporter: Caican Cai Translate overview document into Chinese -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34978) Introduce Asynchronous State APIs
Zakelly Lan created FLINK-34978: --- Summary: Introduce Asynchronous State APIs Key: FLINK-34978 URL: https://issues.apache.org/jira/browse/FLINK-34978 Project: Flink Issue Type: Sub-task Reporter: Zakelly Lan Assignee: Zakelly Lan Fix For: 1.20.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34979) Implement State Future and related utilities
Zakelly Lan created FLINK-34979: --- Summary: Implement State Future and related utilities Key: FLINK-34979 URL: https://issues.apache.org/jira/browse/FLINK-34979 Project: Flink Issue Type: Sub-task Reporter: Zakelly Lan Assignee: Zakelly Lan -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34977) FLIP-433: State Access on DataStream API V2
Weijie Guo created FLINK-34977: -- Summary: FLIP-433: State Access on DataStream API V2 Key: FLINK-34977 URL: https://issues.apache.org/jira/browse/FLINK-34977 Project: Flink Issue Type: Sub-task Components: API / DataStream, API / State Processor Affects Versions: 1.20.0 Reporter: Weijie Guo Assignee: Weijie Guo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34976) LD_PRELOAD environment may not be effective after su to flink user
xiaogang zhou created FLINK-34976: - Summary: LD_PRELOAD environment may not be effective after su to flink user Key: FLINK-34976 URL: https://issues.apache.org/jira/browse/FLINK-34976 Project: Flink Issue Type: New Feature Components: flink-docker Affects Versions: 1.19.0 Reporter: xiaogang zhou -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34975) FLIP-427: ForSt - Disaggregated state Store
Hangxiang Yu created FLINK-34975:
Summary: FLIP-427: ForSt - Disaggregated state Store
Key: FLINK-34975
URL: https://issues.apache.org/jira/browse/FLINK-34975
Project: Flink
Issue Type: New Feature
Components: Runtime / State Backends
Reporter: Hangxiang Yu
Assignee: Hangxiang Yu
This is a sub-FLIP for the disaggregated state management and its related work,
please read the [FLIP-423|https://cwiki.apache.org/confluence/x/R4p3EQ] first
to know the whole story.
As described in FLIP-423, there are some tough issues about embedded state
backend on local file system, respecially when dealing with extremely large
state:
# {*}Constraints of local disk space complicate the prediction of storage
requirements, potentially leading to job failures{*}: Especially in cloud
native deployment mode, pre-allocated local disks typically face strict
capacity constraints, making it challenging to forecast the size requirements
of job states. Over-provisioning disk space results in unnecessary resource
overhead, while under-provisioning risks job failure due to insufficient space.
# *The tight coupling of compute and storage resources leads to
underutilization and increased waste:* Jobs can generally be categorized as
either CPU-intensive or IO-intensive. In a coupled architecture, CPU-intensive
jobs leave a significant portion of storage resources underutilized, whereas
IO-intensive jobs result in idle computing resources.
By considering remote storage as the primary storage, all working states are
maintained on the remote file system, which brings several advantages:
# *Remote storages e.g. S3/HDFS typically offer elastic scalability,
theoretically providing unlimited space.*
# *The allocation of remote storage resources can be optimized by reducing
them for CPU-intensive jobs and augmenting them for IO-intensive jobs, thus
enhancing overall resource utilization.*
# *This architecture facilitates a highly efficient and lightweight process
for checkpointing, recovery, and rescaling through fast copy or simple move.*
This FLIP aims to realize disaggregated state for our new key-value store named
*ForSt* which evloves from RocksDB and supports remote file system. This makes
Flink get rid of the disadvantages by coupled state architecture and embrace
the scalable as well as flexible cloud-native storage.
Please see [FLIP-427 |https://cwiki.apache.org/confluence/x/T4p3EQ]for more
details.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34974) FLIP-424: Asynchronous State APIs
Zakelly Lan created FLINK-34974: --- Summary: FLIP-424: Asynchronous State APIs Key: FLINK-34974 URL: https://issues.apache.org/jira/browse/FLINK-34974 Project: Flink Issue Type: New Feature Components: Runtime / State Backends Reporter: Zakelly Lan Assignee: Zakelly Lan Fix For: 2.0.0 This is a sub-FLIP for the disaggregated state management and its related work, please read the [FLIP-423|https://cwiki.apache.org/confluence/x/R4p3EQ] first to know the whole story. To maximize I/O capacity utilization and enhance the use of pre-allocated computational resources, this FLIP proposes the introduction of asynchronous state APIs. These APIs permit state access to be executed in threads separate from the task thread, returning the result when available. Consequently, the task thread can process another element while awaiting multiple pending state results. This enables concurrent processing of multiple records, ensuring that the latency of individual I/O operations no longer has a direct impact on job performance. This approach is particularly advantageous in scenarios where I/O bandwidth is underutilized and I/O latency is the limiting factor. The Disaggregated Storage Architecture, as discussed in FLIP-423, is a prime example of a scenario characterized by abundant and easily scalable I/O bandwidth coupled with higher I/O latency. The asynchronous state APIs hold great promise for significantly enhancing Flink's performance when dealing with disaggregated state. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34973) FLIP-423: Disaggregated State Storage and Management
Zakelly Lan created FLINK-34973: --- Summary: FLIP-423: Disaggregated State Storage and Management Key: FLINK-34973 URL: https://issues.apache.org/jira/browse/FLINK-34973 Project: Flink Issue Type: New Feature Reporter: Zakelly Lan Fix For: 2.0.0 The past decade has witnessed a dramatic shift in Flink's deployment mode, workload patterns, and hardware improvements. We've moved from the map-reduce era where workers are computation-storage tightly coupled nodes to a cloud-native world where containerized deployments on Kubernetes become standard. To enable Flink's Cloud-Native future, we introduce Disaggregated State Storage and Management that uses DFS as primary storage in Flink 2.0, as promised in the Flink 2.0 Roadmap. Detailed design and story: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=293046855 Also sub-FLIPs: - Asynchronous State APIs ([FLIP-424|https://cwiki.apache.org/confluence/x/SYp3EQ]): Introduce new APIs for asynchronous state access. - Asynchronous Execution Model ([FLIP-425|https://cwiki.apache.org/confluence/x/S4p3EQ]): Implement a non-blocking execution model leveraging the asynchronous APIs introduced in FLIP-424. - Grouping Remote State Access ([FLIP-426|https://cwiki.apache.org/confluence/x/TYp3EQ]): Enable retrieval of remote state data in batches to avoid unnecessary round-trip costs for remote access. - Disaggregated State Store ([FLIP-427|https://cwiki.apache.org/confluence/x/T4p3EQ]): Introduce the initial version of the ForSt disaggregated state store. - Fault Tolerance/Rescale Integration ([FLIP-428|https://cwiki.apache.org/confluence/x/UYp3EQ]): Integrate checkpointing mechanisms with the disaggregated state store for fault tolerance and fast rescaling. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[RESULT][VOTE] FLIP-428: Fault Tolerance/Rescale Integration for Disaggregated State
I'm happy to announce that FLIP-428: Fault Tolerance/Rescale Integration for Disaggregated State [1] has been accepted with 9 approving votes (5 binding) [2]: - Yuan Mei (binding) - Feifan Wang (non-binding) - Rui Fan (binding) - yue ma (no-binding) - Yanfei Lei (binding) - Jeyhun Karimov (no-binding) - gongzhongqiang (no-binding) - Hangxiang Yu (binding) - Yun Tang (binding) There are no disapproving votes. Thanks to everyone who participated in the discussion and voting. [1] https://cwiki.apache.org/confluence/x/UYp3EQ [2] https://lists.apache.org/thread/j7015yymmdy9zp9vr3gg7qpmzjw0o7y2 Best, Jinzhong Li
[RESULT][VOTE] FLIP-424: Asynchronous State APIs
Hi devs, I'm happy to announce that FLIP-424: Asynchronous State APIs [1] has been accepted with 11 approving votes, 7 of which are binding [2]: - Xuannan Su - Yuan Mei (binding) - Piotr Nowojski (binding) - Jing Ge (binding) - Feifan Wang - Rui Fan (binding) - Yunfeng Zhou - Yuepeng Pan - Xintong Song (binding) - Roman Khachatryan (binding) - Yun Tang (binding) [1] https://cwiki.apache.org/confluence/x/SYp3EQ [2] https://lists.apache.org/thread/tplyf17n3409l605bo9promf4o8tvl2j Best, Zakelly
Re: [VOTE] FLIP-428: Fault Tolerance/Rescale Integration for Disaggregated State
+1 (binding) Best Yun Tang From: Hangxiang Yu Sent: Monday, April 1, 2024 10:21 To: dev@flink.apache.org Subject: Re: [VOTE] FLIP-428: Fault Tolerance/Rescale Integration for Disaggregated State +1 (binding) On Mon, Apr 1, 2024 at 9:24 AM gongzhongqiang wrote: > +1(non-binding) > > Best, > > Zhongqiang Gong > > Jinzhong Li 于2024年3月27日周三 19:31写道： > > > Hi devs, > > > > > > I'd like to start a vote on the FLIP-428: Fault Tolerance/Rescale > > Integration for Disaggregated State [1]. The discussion thread is here > [2]. > > > > > > The vote will be open for at least 72 hours unless there is an objection > or > > insufficient votes. > > > > [1] https://cwiki.apache.org/confluence/x/UYp3EQ > > > > [2] https://lists.apache.org/thread/vr8f91p715ct4lop6b3nr0fh4z5p312b > > > > Best, > > > > Jinzhong > > > -- Best, Hangxiang.
Re: [VOTE] FLIP-424: Asynchronous State APIs
+1 (binding) Best Yun Tang From: Zakelly Lan Sent: Monday, April 1, 2024 10:08 To: dev@flink.apache.org Subject: Re: [VOTE] FLIP-424: Asynchronous State APIs Thank you all for the votes! I'm closing this thread and the result will be posted in a separate mail. Best, Zakelly On Sat, Mar 30, 2024 at 4:43 PM Roman Khachatryan wrote: > +1 (binding) > > Regards, > Roman > > > On Fri, Mar 29, 2024 at 7:01 AM Xintong Song > wrote: > > > +1 (binding) > > > > Best, > > > > Xintong > > > > > > > > On Fri, Mar 29, 2024 at 12:51 PM Yuepeng Pan > > wrote: > > > > > +1(non-binding) > > > > > > Best, > > > Yuepeng Pan > > > > > > > > > On 2024/03/29 03:03:53 Yunfeng Zhou wrote: > > > > +1 (non-binding) > > > > > > > > Best, > > > > Yunfeng > > > > > > > > On Wed, Mar 27, 2024 at 6:23 PM Zakelly Lan > > > wrote: > > > > > > > > > > Hi devs, > > > > > > > > > > I'd like to start a vote on the FLIP-424: Asynchronous State APIs > > [1]. > > > The > > > > > discussion thread is here [2]. > > > > > > > > > > The vote will be open for at least 72 hours unless there is an > > > objection or > > > > > insufficient votes. > > > > > > > > > > [1] https://cwiki.apache.org/confluence/x/SYp3EQ > > > > > [2] > https://lists.apache.org/thread/nmd9qd0k8l94ygcfgllxms49wmtz1864 > > > > > > > > > > > > > > > Best, > > > > > Zakelly > > > > > > > > > >
[RESULT][VOTE] FLIP-426: Grouping Remote State Access
I'm happy to announce that FLIP-426: Grouping Remote State Access [1] has been accepted with 8 approving votes (4 binding) [2]: - Yuan Mei (binding) - Feifan Wang (non-binding) - Rui Fan (binding) - gongzhongqiang (no-binding) - Yuepeng Pan (no-binding) - yue ma (no-binding) - Yanfei Lei (binding) - Hangxiang Yu (binding) There are no disapproving votes. Thanks to everyone who participated in the discussion and voting. [1] https://cwiki.apache.org/confluence/x/TYp3EQ [2] https://lists.apache.org/thread/0yvspwhsf0vyvqkkd7snzg33k95v3jbc Best, Jinzhong Li
Re: Re: [ANNOUNCE] Apache Paimon is graduated to Top Level Project
Congratulations! Best, Feng Jin On Mon, Apr 1, 2024 at 10:51 AM weijie guo wrote: > Congratulations! > > Best regards, > > Weijie > > > Hang Ruan 于2024年4月1日周一 09:49写道： > > > Congratulations! > > > > Best, > > Hang > > > > Lincoln Lee 于2024年3月31日周日 00:10写道： > > > > > Congratulations! > > > > > > Best, > > > Lincoln Lee > > > > > > > > > Jark Wu 于2024年3月30日周六 22:13写道： > > > > > > > Congratulations! > > > > > > > > Best, > > > > Jark > > > > > > > > On Fri, 29 Mar 2024 at 12:08, Yun Tang wrote: > > > > > > > > > Congratulations to all Paimon guys! > > > > > > > > > > Glad to see a Flink sub-project has been graduated to an Apache > > > top-level > > > > > project. > > > > > > > > > > Best > > > > > Yun Tang > > > > > > > > > > > > > > > From: Hangxiang Yu > > > > > Sent: Friday, March 29, 2024 10:32 > > > > > To: dev@flink.apache.org > > > > > Subject: Re: Re: [ANNOUNCE] Apache Paimon is graduated to Top Level > > > > Project > > > > > > > > > > Congratulations! > > > > > > > > > > On Fri, Mar 29, 2024 at 10:27 AM Benchao Li > > > > wrote: > > > > > > > > > > > Congratulations! > > > > > > > > > > > > Zakelly Lan 于2024年3月29日周五 10:25写道： > > > > > > > > > > > > > > Congratulations! > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > Zakelly > > > > > > > > > > > > > > On Thu, Mar 28, 2024 at 10:13 PM Jing Ge > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Congrats! > > > > > > > > > > > > > > > > Best regards, > > > > > > > > Jing > > > > > > > > > > > > > > > > On Thu, Mar 28, 2024 at 1:27 PM Feifan Wang < > > zoltar9...@163.com> > > > > > > wrote: > > > > > > > > > > > > > > > > > Congratulations!—— > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > > > > > Feifan Wang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 2024-03-28 20:02:43, "Yanfei Lei" > > > > wrote: > > > > > > > > > >Congratulations! > > > > > > > > > > > > > > > > > > > >Best, > > > > > > > > > >Yanfei > > > > > > > > > > > > > > > > > > > >Zhanghao Chen 于2024年3月28日周四 > > > > 19:59写道： > > > > > > > > > >> > > > > > > > > > >> Congratulations! > > > > > > > > > >> > > > > > > > > > >> Best, > > > > > > > > > >> Zhanghao Chen > > > > > > > > > >> > > > > > > > > > >> From: Yu Li > > > > > > > > > >> Sent: Thursday, March 28, 2024 15:55 > > > > > > > > > >> To: d...@paimon.apache.org > > > > > > > > > >> Cc: dev ; user < > > u...@flink.apache.org > > > > > > > > > > > > > >> Subject: Re: [ANNOUNCE] Apache Paimon is graduated to > Top > > > > Level > > > > > > > > Project > > > > > > > > > >> > > > > > > > > > >> CC the Flink user and dev mailing list. > > > > > > > > > >> > > > > > > > > > >> Paimon originated within the Flink community, initially > > > known > > > > as > > > > > > Flink > > > > > > > > > >> Table Store, and all our incubating mentors are members > of > > > the > > > > > > Flink > > > > > > > > > >> Project Management Committee. I am confident that the > > bonds > > > of > > > > > > > > > >> enduring friendship and close collaboration will > continue > > to > > > > > > unite the > > > > > > > > > >> two communities. > > > > > > > > > >> > > > > > > > > > >> And congratulations all! > > > > > > > > > >> > > > > > > > > > >> Best Regards, > > > > > > > > > >> Yu > > > > > > > > > >> > > > > > > > > > >> On Wed, 27 Mar 2024 at 20:35, Guojun Li < > > > > > gjli.schna...@gmail.com> > > > > > > > > > wrote: > > > > > > > > > >> > > > > > > > > > > >> > Congratulations! > > > > > > > > > >> > > > > > > > > > > >> > Best, > > > > > > > > > >> > Guojun > > > > > > > > > >> > > > > > > > > > > >> > On Wed, Mar 27, 2024 at 5:24 PM wulin < > > > ouyangwu...@163.com> > > > > > > wrote: > > > > > > > > > >> > > > > > > > > > > >> > > Congratulations~ > > > > > > > > > >> > > > > > > > > > > > >> > > > 2024年3月27日 15:54，王刚 > .INVALID> > > > > 写道： > > > > > > > > > >> > > > > > > > > > > > > >> > > > Congratulations~ > > > > > > > > > >> > > > > > > > > > > > > >> > > >> 2024年3月26日 10:25，Jingsong Li < > > jingsongl...@gmail.com > > > > > > > > > 写道： > > > > > > > > > >> > > >> > > > > > > > > > >> > > >> Hi Paimon community, > > > > > > > > > >> > > >> > > > > > > > > > >> > > >> I’m glad to announce that the ASF board has > > approved > > > a > > > > > > > > > resolution to > > > > > > > > > >> > > >> graduate Paimon into a full Top Level Project. > > Thanks > > > > to > > > > > > > > > everyone for > > > > > > > > > >> > > >> your help to get to this point. > > > > > > > > > >> > > >> > > > > > > > > > >> > > >> I just created an issue to track the things we > need > > > to > > > > > > modify > > > > > > > > > [2], > > > > > > > > > >> > > >> please comment on it if you feel that something > is > > > > > > missing. You > > > > > > > > > can > > > > > > > > > >> > > >> refer to apache documentation [1] too. > > > > > > > > > >> > > >> > > > > > > > > >
Re: [DISCUSS] FLIP-428: Fault Tolerance/Rescale Integration for Disaggregated State
Hi Yun, Thanks for your advice. I have added the topic of remote working files cleanup to the current FLIP. Best, Jinzhong On Sat, Mar 30, 2024 at 10:44 AM Yun Tang wrote: > Hi Jinzhong, > > Yes, I know the cleanup mechanism for the remote working directory is the > same as the current Rocksdb state-backend. However, the impact of the > residual files in the remote working directory is different compared with > residual files in the local directory, especially Flink just try the best > to clean up during stateBackend#dispose. > > I agree that we could leave the optimization in the future FLIP, however, > I think we should mention this topic in the current FLIP to make the > overall design more complete and sophisticated. > > > Best > Yun Tang > > From: Jinzhong Li > Sent: Thursday, March 28, 2024 12:45 > To: dev@flink.apache.org > Subject: Re: [DISCUSS] FLIP-428: Fault Tolerance/Rescale Integration for > Disaggregated State > > Hi Feifan, > > Sorry for the misunderstanding. As Hangxiang explained, the basic cleanup > mechanism for remote working directory is the same as rocksdb-statebackend, > that is, when TM exits, forst-statebackend will delete the entire working > dir. Regarding orphaned files cleanup in the case of TM crash, we will > address it in the future FLIP. > > Best, > Jinzhong > > On Thu, Mar 28, 2024 at 12:35 PM Hangxiang Yu wrote: > > > Hi, Yun and Feifan. > > > > Thanks for your reply. > > > > About the cleanup of working dir, as mentioned in FLIP-427, "The life > cycle > > of working dir is managed as before local strategy.". > > Since the current working dir and checkpoint dir are separate, The life > > cycle including creating and cleanup of working dir could be aligned with > > before easily. > > > > On Thu, Mar 28, 2024 at 12:07 PM Feifan Wang wrote: > > > > > And I think the cleanup of working dir should be discussion in > > FLIP-427[1] > > > ( this mail list [2]) ? > > > > > > > > > [1] https://cwiki.apache.org/confluence/x/T4p3EQ > > > [2] https://lists.apache.org/thread/vktfzqvb7t4rltg7fdlsyd9sfdmrc4ft > > > > > > —— > > > > > > Best regards, > > > > > > Feifan Wang > > > > > > > > > > > > > > > At 2024-03-28 11:56:22, "Feifan Wang" wrote: > > > >Hi Jinzhong : > > > > > > > > > > > >> I suggest that we could postpone this topic for now and consider it > > > comprehensively combined with the TM ownership file management in the > > > future FLIP. > > > > > > > > > > > >Sorry I still think we should consider the cleanup of the working dir > in > > > this FLIP, although we may come up with a better solution in a > subsequent > > > flip, I think it is important to maintain the integrity of the current > > > changes. Otherwise we may suffer from wasted DFS space for some time. > > > >Perhaps we only need a simple cleanup strategy at this stage, such as > > > proactive cleanup when TM exits. While this may fail in the case of a > TM > > > crash, it already alleviates the problem. > > > > > > > > > > > > > > > > > > > >—— > > > > > > > >Best regards, > > > > > > > >Feifan Wang > > > > > > > > > > > > > > > > > > > >At 2024-03-28 11:15:11, "Jinzhong Li" > wrote: > > > >>Hi Yun, > > > >> > > > >>Thanks for your reply. > > > >> > > > >>> 1. Why must we have another 'subTask-checkpoint-sub-dir' > > > >>> under the shared directory? if we don't consider making > > > >>> TM ownership in this FLIP, this design seems unnecessary. > > > >> > > > >> Good catch! We will not change the directory layout of shared > > directory > > > in > > > >>this FLIP. I have already removed this part from this FLIP. I think > we > > > >>could revisit this topic in a future FLIP about TM ownership. > > > >> > > > >>> 2. This FLIP forgets to mention the cleanup of the remote > > > >>> working directory in case of the taskmanager crushes, > > > >>> even though this is an open problem, we can still leave > > > >>> some space for future optimization. > > > >> > > > >>Considering that we have plans to merge TM working dir and checkpoint > > dir > > > >>into one directory, I suggest that we could postpone this topic for > now > > > and > > > >>consider it comprehensively combined with the TM ownership file > > > management > > > >>in the future FLIP. > > > >> > > > >>Best, > > > >>Jinzhong > > > >> > > > >> > > > >> > > > >>On Wed, Mar 27, 2024 at 11:49 PM Yun Tang wrote: > > > >> > > > >>> Hi Jinzhong, > > > >>> > > > >>> The overall design looks good. > > > >>> > > > >>> I have two minor questions: > > > >>> > > > >>> 1. Why must we have another 'subTask-checkpoint-sub-dir' under the > > > shared > > > >>> directory? if we don't consider making TM ownership in this FLIP, > > this > > > >>> design seems unnecessary. > > > >>> 2. This FLIP forgets to mention the cleanup of the remote working > > > >>> directory in case of the taskmanager crushes, even though this is > an > > > open > > > >>> problem, we can still leave some space for future
Re: Re: [ANNOUNCE] Apache Paimon is graduated to Top Level Project
Congratulations! Best regards, Weijie Hang Ruan 于2024年4月1日周一 09:49写道： > Congratulations! > > Best, > Hang > > Lincoln Lee 于2024年3月31日周日 00:10写道： > > > Congratulations! > > > > Best, > > Lincoln Lee > > > > > > Jark Wu 于2024年3月30日周六 22:13写道： > > > > > Congratulations! > > > > > > Best, > > > Jark > > > > > > On Fri, 29 Mar 2024 at 12:08, Yun Tang wrote: > > > > > > > Congratulations to all Paimon guys! > > > > > > > > Glad to see a Flink sub-project has been graduated to an Apache > > top-level > > > > project. > > > > > > > > Best > > > > Yun Tang > > > > > > > > > > > > From: Hangxiang Yu > > > > Sent: Friday, March 29, 2024 10:32 > > > > To: dev@flink.apache.org > > > > Subject: Re: Re: [ANNOUNCE] Apache Paimon is graduated to Top Level > > > Project > > > > > > > > Congratulations! > > > > > > > > On Fri, Mar 29, 2024 at 10:27 AM Benchao Li > > > wrote: > > > > > > > > > Congratulations! > > > > > > > > > > Zakelly Lan 于2024年3月29日周五 10:25写道： > > > > > > > > > > > > Congratulations! > > > > > > > > > > > > > > > > > > Best, > > > > > > Zakelly > > > > > > > > > > > > On Thu, Mar 28, 2024 at 10:13 PM Jing Ge > > > > > > > > > > wrote: > > > > > > > > > > > > > Congrats! > > > > > > > > > > > > > > Best regards, > > > > > > > Jing > > > > > > > > > > > > > > On Thu, Mar 28, 2024 at 1:27 PM Feifan Wang < > zoltar9...@163.com> > > > > > wrote: > > > > > > > > > > > > > > > Congratulations!—— > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > > > Feifan Wang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 2024-03-28 20:02:43, "Yanfei Lei" > > > wrote: > > > > > > > > >Congratulations! > > > > > > > > > > > > > > > > > >Best, > > > > > > > > >Yanfei > > > > > > > > > > > > > > > > > >Zhanghao Chen 于2024年3月28日周四 > > > 19:59写道： > > > > > > > > >> > > > > > > > > >> Congratulations! > > > > > > > > >> > > > > > > > > >> Best, > > > > > > > > >> Zhanghao Chen > > > > > > > > >> > > > > > > > > >> From: Yu Li > > > > > > > > >> Sent: Thursday, March 28, 2024 15:55 > > > > > > > > >> To: d...@paimon.apache.org > > > > > > > > >> Cc: dev ; user < > u...@flink.apache.org > > > > > > > > > > > >> Subject: Re: [ANNOUNCE] Apache Paimon is graduated to Top > > > Level > > > > > > > Project > > > > > > > > >> > > > > > > > > >> CC the Flink user and dev mailing list. > > > > > > > > >> > > > > > > > > >> Paimon originated within the Flink community, initially > > known > > > as > > > > > Flink > > > > > > > > >> Table Store, and all our incubating mentors are members of > > the > > > > > Flink > > > > > > > > >> Project Management Committee. I am confident that the > bonds > > of > > > > > > > > >> enduring friendship and close collaboration will continue > to > > > > > unite the > > > > > > > > >> two communities. > > > > > > > > >> > > > > > > > > >> And congratulations all! > > > > > > > > >> > > > > > > > > >> Best Regards, > > > > > > > > >> Yu > > > > > > > > >> > > > > > > > > >> On Wed, 27 Mar 2024 at 20:35, Guojun Li < > > > > gjli.schna...@gmail.com> > > > > > > > > wrote: > > > > > > > > >> > > > > > > > > > >> > Congratulations! > > > > > > > > >> > > > > > > > > > >> > Best, > > > > > > > > >> > Guojun > > > > > > > > >> > > > > > > > > > >> > On Wed, Mar 27, 2024 at 5:24 PM wulin < > > ouyangwu...@163.com> > > > > > wrote: > > > > > > > > >> > > > > > > > > > >> > > Congratulations~ > > > > > > > > >> > > > > > > > > > > >> > > > 2024年3月27日 15:54，王刚 .INVALID> > > > 写道： > > > > > > > > >> > > > > > > > > > > > >> > > > Congratulations~ > > > > > > > > >> > > > > > > > > > > > >> > > >> 2024年3月26日 10:25，Jingsong Li < > jingsongl...@gmail.com > > > > > > > 写道： > > > > > > > > >> > > >> > > > > > > > > >> > > >> Hi Paimon community, > > > > > > > > >> > > >> > > > > > > > > >> > > >> I’m glad to announce that the ASF board has > approved > > a > > > > > > > > resolution to > > > > > > > > >> > > >> graduate Paimon into a full Top Level Project. > Thanks > > > to > > > > > > > > everyone for > > > > > > > > >> > > >> your help to get to this point. > > > > > > > > >> > > >> > > > > > > > > >> > > >> I just created an issue to track the things we need > > to > > > > > modify > > > > > > > > [2], > > > > > > > > >> > > >> please comment on it if you feel that something is > > > > > missing. You > > > > > > > > can > > > > > > > > >> > > >> refer to apache documentation [1] too. > > > > > > > > >> > > >> > > > > > > > > >> > > >> And, we already completed the GitHub repo migration > > > [3], > > > > > please > > > > > > > > update > > > > > > > > >> > > >> your local git repo to track the new repo [4]. > > > > > > > > >> > > >> > > > > > > > > >> > > >> You can run the following command to complete the > > > remote > > > > > repo > > > > > > > > tracking > > > > > > > > >> > > >> migration. > > > > > > > > >> > > >> > > > > > > > > >> > > >> git remote set-url
Re: [VOTE] FLIP-426: Grouping Remote State Access
+1 (binding) On Sun, Mar 31, 2024 at 9:02 PM Yanfei Lei wrote: > +1 (binding) > > Best, > Yanfei > > yue ma 于2024年3月29日周五 16:10写道： > > > > +1 (non-binding) > > > > Jinzhong Li 于2024年3月27日周三 18:57写道： > > > > > Hi devs, > > > > > > I'd like to start a vote on the FLIP-426: Grouping Remote State Access > [1]. > > > The discussion thread is here [2]. > > > > > > The vote will be open for at least 72 hours unless there is an > objection or > > > insufficient votes. > > > > > > > > > [1] https://cwiki.apache.org/confluence/x/TYp3EQ > > > > > > [2] https://lists.apache.org/thread/bt931focfl9971cwq194trmf3pkdsxrf > > > > > > > > > Best, > > > > > > Jinzhong > > > > > > > > > -- > > Best, > > Yue > -- Best, Hangxiang.
[RESULT][VOTE] FLIP-427: Disaggregated state Store
I'm happy to announce that FLIP-427: Disaggregated state Store[1] has been accepted with 7 approving votes (4 binding) [2]: - Yuan Mei (binding) - Feifan Wang (non-binding) - Piotr Nowojski (binding) - Rui Fan (binding) - Yun Tang (binding) - Yuepeng Pan (non-binding) - yue ma (non-binding) There are no disapproving votes. Thanks to everyone who participated in the discussion and voting. [1] https://cwiki.apache.org/confluence/x/T4p3EQ [2] https://lists.apache.org/thread/lqpovwx3kpszgfcgl0olfhxotfo9d6nz Best, Hangxiang
Re: [VOTE] FLIP-427: Disaggregated state Store
Thanks all for the votes! I'm closing the vote and the result will be posted in a separate mail. On Fri, Mar 29, 2024 at 4:29 PM yue ma wrote: > +1(non-binding) > > Hangxiang Yu 于2024年3月27日周三 18:37写道： > > > Hi devs, > > > > Thanks all for your valuable feedback about FLIP-427: Disaggregated state > > Store [1]. > > I'd like to start a vote on it. The discussion thread is here [2]. > > > > The vote will be open for at least 72 hours unless there is an objection > or > > insufficient votes. > > > > [1] https://cwiki.apache.org/confluence/x/T4p3EQ > > [2] https://lists.apache.org/thread/vktfzqvb7t4rltg7fdlsyd9sfdmrc4ft > > > > > > Best, > > Hangxiang > > > > > -- > Best, > Yue > -- Best, Hangxiang.
Re: [VOTE] FLIP-428: Fault Tolerance/Rescale Integration for Disaggregated State
+1 (binding) On Mon, Apr 1, 2024 at 9:24 AM gongzhongqiang wrote: > +1(non-binding) > > Best, > > Zhongqiang Gong > > Jinzhong Li 于2024年3月27日周三 19:31写道： > > > Hi devs, > > > > > > I'd like to start a vote on the FLIP-428: Fault Tolerance/Rescale > > Integration for Disaggregated State [1]. The discussion thread is here > [2]. > > > > > > The vote will be open for at least 72 hours unless there is an objection > or > > insufficient votes. > > > > [1] https://cwiki.apache.org/confluence/x/UYp3EQ > > > > [2] https://lists.apache.org/thread/vr8f91p715ct4lop6b3nr0fh4z5p312b > > > > Best, > > > > Jinzhong > > > -- Best, Hangxiang.
Re: [DISCUSS] FLIP-427: Disaggregated State Store
Hi Yun. Thanks for the great suggestion. I just added related information into the FLIP. On Sat, Mar 30, 2024 at 10:49 AM Yun Tang wrote: > Hi Feifan, > > I just replied in the discussion of FLIP-428. I agree that we could leave > the clean-up optimization in the future FLIP, however, I think we should > mention this topic explicitly in the current FLIP to make the overall > design complete and more sophisticated. > > Best > Yun Tang > > From: Feifan Wang > Sent: Thursday, March 28, 2024 12:35 > To: dev@flink.apache.org > Subject: Re: [DISCUSS] FLIP-427: Disaggregated State Store > > Thanks for your reply, Hangxiang. I totally agree with you about the jni > part. > > Hi Yun Tang, I just noticed that FLIP-427 mentions “The life cycle of > working dir is managed as before local strategy.” IIUC, the working dir > will be deleted after TaskManager exit. And I think that's enough for > current stage, WDYT ? > > —— > > Best regards, > > Feifan Wang > > > > > At 2024-03-28 12:18:56, "Hangxiang Yu" wrote: > >Hi, Feifan. > > > >Thanks for your reply. > > > >What if we only use jni to access DFS that needs to reuse Flink > FileSystem? > >> And all local disk access through native api. This idea is based on the > >> understanding that jni overhead is not worth mentioning compared to DFS > >> access latency. It might make more sense to consider avoiding jni > overhead > >> for faster local disks. Since local disk as secondary is already under > >> consideration [1], maybe we can discuss in that FLIP whether to use > native > >> api to access local disk? > >> > >This is a good suggestion. It's reasonable to use native api to access > >local disk cache since it requires lower latency compared to remote > access. > >I also believe that the jni overhead is relatively negligible when weighed > >against the latency of remote I/O as mentioned in the FLIP. > >So I think we could just go on proposal 2 and keep proposal 1 as a > >potential future optimization, which could work better when there is a > >higher performance requirement or some native libraries of filesystems > have > >significantly higher performance and resource usage compared to their java > >libs. > > > > > >On Thu, Mar 28, 2024 at 11:39 AM Feifan Wang wrote: > > > >> Thanks for this valuable proposal Hangxiang ! > >> > >> > >> > If we need to introduce a JNI call during each filesystem call, that > >> would be N times JNI cost compared with the current RocksDB > state-backend's > >> JNI cost. > >> What if we only use jni to access DFS that needs to reuse Flink > >> FileSystem? And all local disk access through native api. This idea is > >> based on the understanding that jni overhead is not worth mentioning > >> compared to DFS access latency. It might make more sense to consider > >> avoiding jni overhead for faster local disks. Since local disk as > secondary > >> is already under consideration [1], maybe we can discuss in that FLIP > >> whether to use native api to access local disk? > >> > >> > >> >I'd suggest keeping `state.backend.forSt.working-dir` as it is for now. > >> >Different disaggregated state storages may have their own semantics > about > >> >this configuration, e.g. life cycle, supported file systems or > storages. > >> I agree with considering moving this configuration up to the engine > level > >> until there are other disaggreated backends. > >> > >> > >> [1] https://cwiki.apache.org/confluence/x/U4p3EQ > >> > >> —— > >> > >> Best regards, > >> > >> Feifan Wang > >> > >> > >> > >> > >> At 2024-03-28 09:55:48, "Hangxiang Yu" wrote: > >> >Hi, Yun. > >> >Thanks for the reply. > >> > > >> >The JNI cost you considered is right. As replied to Yue, I agreed to > leave > >> >space and consider proposal 1 as an optimization in the future, which > is > >> >also updated in the FLIP. > >> > > >> >The other question is that the configuration of > >> >> `state.backend.forSt.working-dir` looks too coupled with the ForSt > >> >> state-backend, how would it be if we introduce another disaggregated > >> state > >> >> storage? Thus, I think `state.backend.disaggregated.working-dir` > might > >> be a > >> >> better configuration name. > >> > > >> >I'd suggest keeping `state.backend.forSt.working-dir` as it is for now. > >> >Different disaggregated state storages may have their own semantics > about > >> >this configuration, e.g. life cycle, supported file systems or > storages. > >> >Maybe it's more suitable to consider it together when we introduce > other > >> >disaggregated state storages in the future. > >> > > >> >On Thu, Mar 28, 2024 at 12:02 AM Yun Tang wrote: > >> > > >> >> Hi Hangxiang, > >> >> > >> >> The design looks good, and I also support leaving space for proposal > 1. > >> >> > >> >> As you know, loading index/filter/data blocks for querying across > levels > >> >> would introduce high IO access within the LSM tree for old data. If > we > >> need > >> >> to introduce a JNI call
Re: [VOTE] FLIP-424: Asynchronous State APIs
Thank you all for the votes! I'm closing this thread and the result will be posted in a separate mail. Best, Zakelly On Sat, Mar 30, 2024 at 4:43 PM Roman Khachatryan wrote: > +1 (binding) > > Regards, > Roman > > > On Fri, Mar 29, 2024 at 7:01 AM Xintong Song > wrote: > > > +1 (binding) > > > > Best, > > > > Xintong > > > > > > > > On Fri, Mar 29, 2024 at 12:51 PM Yuepeng Pan > > wrote: > > > > > +1(non-binding) > > > > > > Best, > > > Yuepeng Pan > > > > > > > > > On 2024/03/29 03:03:53 Yunfeng Zhou wrote: > > > > +1 (non-binding) > > > > > > > > Best, > > > > Yunfeng > > > > > > > > On Wed, Mar 27, 2024 at 6:23 PM Zakelly Lan > > > wrote: > > > > > > > > > > Hi devs, > > > > > > > > > > I'd like to start a vote on the FLIP-424: Asynchronous State APIs > > [1]. > > > The > > > > > discussion thread is here [2]. > > > > > > > > > > The vote will be open for at least 72 hours unless there is an > > > objection or > > > > > insufficient votes. > > > > > > > > > > [1] https://cwiki.apache.org/confluence/x/SYp3EQ > > > > > [2] > https://lists.apache.org/thread/nmd9qd0k8l94ygcfgllxms49wmtz1864 > > > > > > > > > > > > > > > Best, > > > > > Zakelly > > > > > > > > > >
Re: Re: [ANNOUNCE] Apache Paimon is graduated to Top Level Project
Congratulations! Best, Hang Lincoln Lee 于2024年3月31日周日 00:10写道： > Congratulations! > > Best, > Lincoln Lee > > > Jark Wu 于2024年3月30日周六 22:13写道： > > > Congratulations! > > > > Best, > > Jark > > > > On Fri, 29 Mar 2024 at 12:08, Yun Tang wrote: > > > > > Congratulations to all Paimon guys! > > > > > > Glad to see a Flink sub-project has been graduated to an Apache > top-level > > > project. > > > > > > Best > > > Yun Tang > > > > > > > > > From: Hangxiang Yu > > > Sent: Friday, March 29, 2024 10:32 > > > To: dev@flink.apache.org > > > Subject: Re: Re: [ANNOUNCE] Apache Paimon is graduated to Top Level > > Project > > > > > > Congratulations! > > > > > > On Fri, Mar 29, 2024 at 10:27 AM Benchao Li > > wrote: > > > > > > > Congratulations! > > > > > > > > Zakelly Lan 于2024年3月29日周五 10:25写道： > > > > > > > > > > Congratulations! > > > > > > > > > > > > > > > Best, > > > > > Zakelly > > > > > > > > > > On Thu, Mar 28, 2024 at 10:13 PM Jing Ge > > > > > > > wrote: > > > > > > > > > > > Congrats! > > > > > > > > > > > > Best regards, > > > > > > Jing > > > > > > > > > > > > On Thu, Mar 28, 2024 at 1:27 PM Feifan Wang > > > > wrote: > > > > > > > > > > > > > Congratulations!—— > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > Feifan Wang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 2024-03-28 20:02:43, "Yanfei Lei" > > wrote: > > > > > > > >Congratulations! > > > > > > > > > > > > > > > >Best, > > > > > > > >Yanfei > > > > > > > > > > > > > > > >Zhanghao Chen 于2024年3月28日周四 > > 19:59写道： > > > > > > > >> > > > > > > > >> Congratulations! > > > > > > > >> > > > > > > > >> Best, > > > > > > > >> Zhanghao Chen > > > > > > > >> > > > > > > > >> From: Yu Li > > > > > > > >> Sent: Thursday, March 28, 2024 15:55 > > > > > > > >> To: d...@paimon.apache.org > > > > > > > >> Cc: dev ; user > > > > > > > > >> Subject: Re: [ANNOUNCE] Apache Paimon is graduated to Top > > Level > > > > > > Project > > > > > > > >> > > > > > > > >> CC the Flink user and dev mailing list. > > > > > > > >> > > > > > > > >> Paimon originated within the Flink community, initially > known > > as > > > > Flink > > > > > > > >> Table Store, and all our incubating mentors are members of > the > > > > Flink > > > > > > > >> Project Management Committee. I am confident that the bonds > of > > > > > > > >> enduring friendship and close collaboration will continue to > > > > unite the > > > > > > > >> two communities. > > > > > > > >> > > > > > > > >> And congratulations all! > > > > > > > >> > > > > > > > >> Best Regards, > > > > > > > >> Yu > > > > > > > >> > > > > > > > >> On Wed, 27 Mar 2024 at 20:35, Guojun Li < > > > gjli.schna...@gmail.com> > > > > > > > wrote: > > > > > > > >> > > > > > > > > >> > Congratulations! > > > > > > > >> > > > > > > > > >> > Best, > > > > > > > >> > Guojun > > > > > > > >> > > > > > > > > >> > On Wed, Mar 27, 2024 at 5:24 PM wulin < > ouyangwu...@163.com> > > > > wrote: > > > > > > > >> > > > > > > > > >> > > Congratulations~ > > > > > > > >> > > > > > > > > > >> > > > 2024年3月27日 15:54，王刚 > > 写道： > > > > > > > >> > > > > > > > > > > >> > > > Congratulations~ > > > > > > > >> > > > > > > > > > > >> > > >> 2024年3月26日 10:25，Jingsong Li > > > > 写道： > > > > > > > >> > > >> > > > > > > > >> > > >> Hi Paimon community, > > > > > > > >> > > >> > > > > > > > >> > > >> I’m glad to announce that the ASF board has approved > a > > > > > > > resolution to > > > > > > > >> > > >> graduate Paimon into a full Top Level Project. Thanks > > to > > > > > > > everyone for > > > > > > > >> > > >> your help to get to this point. > > > > > > > >> > > >> > > > > > > > >> > > >> I just created an issue to track the things we need > to > > > > modify > > > > > > > [2], > > > > > > > >> > > >> please comment on it if you feel that something is > > > > missing. You > > > > > > > can > > > > > > > >> > > >> refer to apache documentation [1] too. > > > > > > > >> > > >> > > > > > > > >> > > >> And, we already completed the GitHub repo migration > > [3], > > > > please > > > > > > > update > > > > > > > >> > > >> your local git repo to track the new repo [4]. > > > > > > > >> > > >> > > > > > > > >> > > >> You can run the following command to complete the > > remote > > > > repo > > > > > > > tracking > > > > > > > >> > > >> migration. > > > > > > > >> > > >> > > > > > > > >> > > >> git remote set-url origin > > > > https://github.com/apache/paimon.git > > > > > > > >> > > >> > > > > > > > >> > > >> If you have a different name, please change the > > 'origin' > > > to > > > > > > your > > > > > > > remote > > > > > > > >> > > name. > > > > > > > >> > > >> > > > > > > > >> > > >> Please join me in celebrating! > > > > > > > >> > > >> > > > > > > > >> > > >> [1] > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > https://incubator.apache.org/guides/transferring.html#life_after_graduation > > > > > > >
Re: Bug report for reading Hive table as streaming source.
Thanks for reporting. Could you please help create a jira about it?
Best regards,
Yuxia
- 原始邮件 -
发件人: "Xiaolong Wang"
收件人: "dev"
发送时间: 星期四, 2024年 3 月 28日 下午 5:11:20
主题: Re: Bug report for reading Hive table as streaming source.
I think it worth mentioning in the documentation of Hive read that it
cannot read a table that has more than 32,767 partitions.
On Thu, Mar 28, 2024 at 5:10 PM Xiaolong Wang
wrote:
> Found out the reason:
>
> It turned out that in Flink, it uses hive’s IMetaStoreClient to fetch
> partitions using the following method:
>
> List listPartitionNames(String db_name, String tbl_name,
> short max_parts) throws MetaException, TException;
>
> where the max_parts represents the max number of partitions it can fetch
> from the Hive metastore.
> So the max number of partitions it can fetch is Short.MAX_VALUE, which is
> 32767 .
>
> But the table has a way more partition number than the max value, thus the
> list partition operations cannot fetch all partitions, hence it cannot
> consume the recent partition.
>
> On Tue, Mar 26, 2024 at 5:00 PM Xiaolong Wang
> wrote:
>
>> Hi,
>>
>> I found a weird bug when reading a Hive table as a streaming source.
>>
>> In summary, if the first partition is not time related, then the Hive
>> table cannot be read as a streaming source.
>>
>> e.g.
>>
>> I've a Hive table in the definition of
>>
>> ```
>> CREATE TABLE article (
>> id BIGINT,
>> edition STRING,
>> dt STRING,
>> hh STRING
>> )
>> PARTITIONED BY (edition, dt, hh)
>> USING orc;
>> ```
>> Then I try to query it as a streaming source:
>>
>> ```
>> INSERT INTO kafka_sink
>> SELECT id
>> FROM article /*+ OPTIONS('streaming-source.enable' = 'true',
>> 'streaming-source.partition-order' = 'partition-name',
>> 'streaming-source.consume-start-offset' =
>> 'edition=en_US/dt=2024-03-26/hh=00') */
>> ```
>>
>> And I see no output in the `kafka_sink`.
>>
>> Then I defined an external table pointing to the same path but has no
>> `edition` partition,
>>
>> ```
>> CREATE TABLE en_article (
>> id BIGINT,
>> edition STRING,
>> dt STRING,
>> hh STRING
>> )
>> PARTITIONED BY (edition, dt, hh)
>> LOCATION 's3://xxx/article/edition=en_US'
>> USING orc;
>> ```
>>
>> And insert with the following statement:
>>
>> ```
>> INSERT INTO kafka_sink
>> SELECT id
>> FROM en_article /*+ OPTIONS('streaming-source.enable' = 'true',
>> 'streaming-source.partition-order' = 'partition-name',
>> 'streaming-source.consume-start-offset' = 'dt=2024-03-26/hh=00') */
>> ```
>>
>> The data is sinked.
>>
>>
>
Re: [VOTE] FLIP-428: Fault Tolerance/Rescale Integration for Disaggregated State
+1(non-binding) Best, Zhongqiang Gong Jinzhong Li 于2024年3月27日周三 19:31写道： > Hi devs, > > > I'd like to start a vote on the FLIP-428: Fault Tolerance/Rescale > Integration for Disaggregated State [1]. The discussion thread is here [2]. > > > The vote will be open for at least 72 hours unless there is an objection or > insufficient votes. > > [1] https://cwiki.apache.org/confluence/x/UYp3EQ > > [2] https://lists.apache.org/thread/vr8f91p715ct4lop6b3nr0fh4z5p312b > > Best, > > Jinzhong >
Re: [VOTE] FLIP-428: Fault Tolerance/Rescale Integration for Disaggregated State
+1 (non-binding) Regards, Jeyhun On Wed, Mar 27, 2024 at 12:31 PM Jinzhong Li wrote: > Hi devs, > > > I'd like to start a vote on the FLIP-428: Fault Tolerance/Rescale > Integration for Disaggregated State [1]. The discussion thread is here [2]. > > > The vote will be open for at least 72 hours unless there is an objection or > insufficient votes. > > [1] https://cwiki.apache.org/confluence/x/UYp3EQ > > [2] https://lists.apache.org/thread/vr8f91p715ct4lop6b3nr0fh4z5p312b > > Best, > > Jinzhong >
[jira] [Created] (FLINK-34972) MigrationTestsSnapshotGenerator does not work properly for all subclasses of SnapshotMigrationTestBase
lincoln lee created FLINK-34972:
---
Summary: MigrationTestsSnapshotGenerator does not work properly
for all subclasses of SnapshotMigrationTestBase
Key: FLINK-34972
URL: https://issues.apache.org/jira/browse/FLINK-34972
Project: Flink
Issue Type: Bug
Affects Versions: 1.19.0
Reporter: lincoln lee
This issue was found on the followup work of 1.19.0 releasing
https://issues.apache.org/jira/browse/FLINK-34712.
{code}MigrationTestsSnapshotGenerator{code} actually didn't generate the
corresponding testing files for the new version(1.19) without any abnormal
logs. And only appears in all subclasses of
{code}SnapshotMigrationTestBase{code}.
more context: https://github.com/apache/flink/pull/24517
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[RESULT][VOTE] FLIP-425: Asynchronous Execution Model
I'm happy to announce that FLIP-425: Asynchronous Execution Model[1] has been accepted with 10 approving votes (6 binding) [2]: - Xuannan Su (non-binding) - Yuan Mei (binding) - Piotr Nowojski (binding) - Feifan Wang (non-binding) - Jing Ge (binding) - Rui Fan (binding) - Yunfeng Zhou (non-binding) - Xintong Song (binding) - Yue Ma (non-binding) - Roman Khachatryan (binding) There are no disapproving votes. Thanks to everyone who participated in the discussion and voting. [1] https://cwiki.apache.org/confluence/x/S4p3EQ [2] https://lists.apache.org/thread/drglfqv83z5n4hybdjn0lvcc3xjkrtk0 Best regards, Yanfei
Re: [VOTE] FLIP-428: Fault Tolerance/Rescale Integration for Disaggregated State
+1 (binding) Best, Yanfei yue ma 于2024年3月29日周五 17:29写道： > > +1(non-binding) > > Jinzhong Li 于2024年3月27日周三 19:31写道： > > > Hi devs, > > > > > > I'd like to start a vote on the FLIP-428: Fault Tolerance/Rescale > > Integration for Disaggregated State [1]. The discussion thread is here [2]. > > > > > > The vote will be open for at least 72 hours unless there is an objection or > > insufficient votes. > > > > [1] https://cwiki.apache.org/confluence/x/UYp3EQ > > > > [2] https://lists.apache.org/thread/vr8f91p715ct4lop6b3nr0fh4z5p312b > > > > Best, > > > > Jinzhong > > > > > -- > Best, > Yue
Re: [VOTE] FLIP-426: Grouping Remote State Access
+1 (binding) Best, Yanfei yue ma 于2024年3月29日周五 16:10写道： > > +1 (non-binding) > > Jinzhong Li 于2024年3月27日周三 18:57写道： > > > Hi devs, > > > > I'd like to start a vote on the FLIP-426: Grouping Remote State Access [1]. > > The discussion thread is here [2]. > > > > The vote will be open for at least 72 hours unless there is an objection or > > insufficient votes. > > > > > > [1] https://cwiki.apache.org/confluence/x/TYp3EQ > > > > [2] https://lists.apache.org/thread/bt931focfl9971cwq194trmf3pkdsxrf > > > > > > Best, > > > > Jinzhong > > > > > -- > Best, > Yue
[jira] [Created] (FLINK-34971) Add the function of History Server cleaning job archives regularly
dengxaing created FLINK-34971: - Summary: Add the function of History Server cleaning job archives regularly Key: FLINK-34971 URL: https://issues.apache.org/jira/browse/FLINK-34971 Project: Flink Issue Type: New Feature Reporter: dengxaing Currently, flink does not provide the ability to regularly clean the archives of completed jobs. I think this ability is necessary. When flink provides this ability, users can decide whether to use this ability. They also can define the time for regular cleaning. -- This message was sent by Atlassian Jira (v8.20.10#820010)
