Hi Jark! Thank you for your insightful suggestions. This FIP is a small step for Fluss towards multi-remote (or multi-cloud) storage. As you mentioned, we envision future support for commit-level multi-pathing, similar to the approaches taken by Paimon and Lance.
For your comments on the current FIP. I'm generally in agreement. 1. FileSystem#obtainSecurityToken(FsPath f) For the current implementation, obtainSecurityToken(FsPath f) is actually redundant and can be removed. 2. GetFileSystemSecurityTokenRequest/Response and Client Token Management Your suggestion simplifies the implementation of multi-path authorization. By deferring table-level authentication to a later stage, we can expedite the landing of this FIP. I will update the FIP accordingly. Best regards, Liebing Yu On Tue, 3 Mar 2026 at 01:19, Jark Wu <[email protected]> wrote: > Hi Liebing, > > Thank you for the proposal. I believe this is an excellent initiative > to improve throughput for large-scale clusters utilizing remote > storage. > > The current design implements multi-location support at the table or > partition level, meaning only new tables and partitions will utilize > new remote locations. Consequently, even after upgrading the cluster > to support multiple paths, data distribution will remain concentrated > in a single location for an extended period, failing to achieve rapid > traffic fan-out. In contrast, industry solutions like Paimon support > "data-file.external-paths" [1] to distribute new data files across > multiple paths, and Lance has recently introduced a file-level > multi-base layout [2]. > > Ultimately, we need file-level multi-location support (I believe this > approach will resolve most of the concerns raised above by Yang Guo). > However, I am fine with supporting partition-level multi-location as > an initial phase, provided we have a clear roadmap toward the final > solution. > > Regarding the design details of this FIP, I have the following comments: > > 1. FileSystem#obtainSecurityToken(FsPath f) > We should not add the FsPath parameter to the obtainSecurityToken > interface for now. Because in current design, this interface only > retrieve the security token for the entire filesystem rather than for > a specific path. Since a filesystem is defined per authority, the > authority does not need to be derived from an FsPath. > > In fact, we plan to refactor the Filesystem soon. This refactoring > will add the FsPath parameter to obtainSecurityToken, ensuring the > returned token is strictly scoped to that specific path. This change > aims to address current permission leakage issues where a token > requested for reading one table inadvertently grants access to all > remote files of other tables. > > 2. GetFileSystemSecurityTokenRequest/Response and Client Token Management > > Current Issue: The FIP proposes maintaining a SecurityTokenManager per > LogScanner. However, since tokens are shared at the filesystem > granularity, tokens for the same FsKey across different tables should > be consolidated. Therefore, the DefaultSecurityTokenManager must be > maintained within the FlussConnection; otherwise, > SecurityTokenManagers for different tables will overwrite each other's > tokens. > > Recommendation: A straightforward approach is to leave > GetFileSystemSecurityTokenRequest unchanged while modifying > GetFileSystemSecurityTokenResponse to return a list of tokens. The > server side would then return STS tokens for each FsKey configured in > the cluster. The client-side Filesystem would subsequently retrieve > the corresponding STS token based on the FsKey. This avoids changes to > the LogScanner logic. > > While this approach retains the existing permission leakage issue, > that problem is already present today. We can address it in a > separate, dedicated FIP to simplify the scope and implementation of > the current proposal. > > Best, > Jark > > [1] https://paimon.apache.org/docs/1.3/maintenance/configurations/ > [2] > https://lancedb.com/blog/rethinking-table-file-paths-lance-multi-base-layout/ > > > On Sat, 28 Feb 2026 at 20:37, Yang Guo <[email protected]> wrote: > > > > Hi Liebing and all, > > > > This is a good FIP to resolve bottlenecks in the remote storage. Thanks > for > > your effort. The design looks good to me and the above discussion has > > covered some concerns in my mind. > > > > Now there are some further considerations I'm thinking of: > > > > 1. What happens if a path goes down? > > Right now, there’s no automatic failover. If one S3 bucket (or HDFS > path) > > dies, every table or partition assigned to it just fails. Could we add > > simple health checks? If a path looks dead, the remote dir selector > > temporarily skips it until it’s back up. > > > > 2. New paths don't always help old data. > > The routing only happens when a new table or new partition is created. > And > > it depends on the partition strategy. > > - If the table is using time-based partitions (e.g., daily), adding new > > paths works well because new data goes to new partitions on new paths. > > - But for non-partitioned tables, or if it keeps writing to old > partitions, > > the new paths sit idle. The traffic never shifts over. > > It requires developers to think further about partition strategy and > input > > data when adding remote dirs. > > > > 3. Managing "weights" is tricky manually for developers/maintainers. > > Since the weighted round-robin is static: > > - Developers/Maintainers have to determine the right weights based on > > current traffic. > > - If you skew weights to favor a path, you have to remember to > > rebalance them later, or that path gets overloaded forever. E.g. If two > > paths are weighted [1, 2] in the beginning to rebalance the higher > traffic > > in the first path. Developers/Maintainers should remember to change the > > weights back to [1, 1] after the traffic is balanced between two paths. > > Otherwise the traffic in the second path will keep growing. > > - Also, setting a weight to 0 behaves differently depending on your > > partition type (time-based paths eventually go quiet, but field-based > ones > > like "country=US" keep writing there forever). > > Instead of manual tuning, could we eventually make this dynamic? Let the > > system adjust weights based on real-time latency or throttling metrics. > > > > The points above are about future operational considerations—regarding > > failover and maintenance after this solution is deployed. I think they > > won't block this FIP. We may not need to fix these right now. Just bring > > them into this discussion. > > > > Regards, > > Yang Guo > > > > On Fri, Feb 27, 2026 at 5:53 PM Liebing Yu <[email protected]> wrote: > > > > > Hi Lorenzo, sorry for the late reply. > > > > > > Thanks for the AWS example! This further solidifies the case for > multi-path > > > support. > > > > > > Regarding your question about multi-cloud support: > > > Our current design naturally supports multi-cloud object storage > systems. > > > Since the implementation is built upon a multi-schema filesystem > > > abstraction (supporting schemes like s3://, oss://, abfs://, etc.), the > > > system is inherently "cloud-agnostic." > > > > > > Best regards, > > > Liebing Yu > > > > > > > > > On Wed, 4 Feb 2026 at 23:37, Lorenzo Affetti via dev < > [email protected] > > > > > > > wrote: > > > > > > > This is quite an interesting FIP and I think it is a significant > > > > enhancement, especially for large-scale clusters. > > > > > > > > I think you can also add the AWS case in your motivation: > > > > > > > > > > > > https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-design-patterns.html#optimizing-performance-high-request-rate > > > > AWS automatically scales if requests exceed 5,500 per second for the > same > > > > prefix, which results in transient 503 errors. > > > > Your approach would eliminate this problem by providing another > bucket. > > > > > > > > I was wondering if it might also provide the possibility of > configuring > > > the > > > > same Fluss cluster for multi-cloud object storage systems. > > > > From a design perspective, nothing should prevent me from storing > remote > > > > data on both Azure and AWS at the same time, probably resulting in > > > > different performance numbers for different partitions/tables. > > > > Should the design force the use of only 1 filesystem implementation? > > > > > > > > Thank you again! > > > > > > > > On Fri, Jan 30, 2026 at 7:59 AM Liebing Yu <[email protected]> > wrote: > > > > > > > > > Hi Yuxia, thanks for the thoughtful response. Let me go through > your > > > > > questions one by one. > > > > > > > > > > 1. I think after we support `remote.data.dirs`, different schemas > will > > > be > > > > > supported naturally. > > > > > 2. Yes, I think we should change from `PbTablePath` to > > > > > `PbPhysicalTablePath`. > > > > > 3. Thanks for the reminder. I'll poc authentication in > > > > > https://github.com/apache/fluss/issues/2518. But it doesn't block > the > > > > > multiple-paths implementation in Fluss server in > > > > > https://github.com/apache/fluss/issues/2517. > > > > > 4. For a partition table, the table itself has a remote data dir > for > > > > > metadata (such as lake offset). And each partition has its own > remote > > > dir > > > > > for table data (e.g. kv or log data). > > > > > 5. Legacy clients can access data in the new cluster. > > > > > > > > > > - If the permissions of the paths specified in > `remote.data.dirs` on > > > > the > > > > > new cluster match those configured in `remote.data.dir`, > seamless > > > > > access is > > > > > achievable. > > > > > - If the permissions are inconsistent, access permissions must > be > > > > > explicitly configured. For example, when using OSS, a policy > > > granting > > > > > access permissions to the account identified by `fs.oss.roleArn` > > > must > > > > be > > > > > configured for each bucket specified in `remote.data.dirs`. > > > > > > > > > > > > > > > Best regards, > > > > > Liebing Yu > > > > > > > > > > > > > > > On Thu, 29 Jan 2026 at 10:07, Yuxia Luo <[email protected]> wrote: > > > > > > > > > > > Hi, Liebing > > > > > > > > > > > > Thanks for the detailed FIP. I have a few questions: > > > > > > 1. Does `remote.data.dirs` support paths with different schemes? > For > > > > > > example: > > > > > > ``` > > > > > > remote.data.dirs: oss://bucket1/fluss-data, > s3://bucket2/fluss-data > > > > > > ``` > > > > > > > > > > > > 2. Should `GetFileSystemSecurityTokenRequest` include partition? > > > > > > The FIP adds `table_path` to the request, but since different > > > > partitions > > > > > > may reside on different remote paths (and require different > tokens), > > > > > > should the request also include partition information? > > > > > > > > > > > > 3. Just a reminder that `DefaultSecurityTokenManager` will become > > > more > > > > > > complex... > > > > > > This is not a blocker, but worth a poc to recoginize any > complexity > > > > > > > > > > > > 4. I want to confirm my understanding: For a partitioned table, > does > > > > the > > > > > > table itself have a remote dir, AND each partition also has its > own > > > > > remote > > > > > > dir? > > > > > > > > > > > > Or is it: > > > > > > - Non-partitioned table → table-level remote dir > > > > > > - Partitioned table → only partition-level remote dirs (no > > > > table-level)? > > > > > > > > > > > > 5. Can old clients (without table path in token request) still > read > > > > data > > > > > > from new clusters? > > > > > > One possibe solution is : For RPCs without table information, the > > > > server > > > > > > returns a token for the first dir in `remote.data.dirs`. Or other > > > ways > > > > > that > > > > > > allow users to configure the cluster to keep compatibility > > > > > > > > > > > > > > > > > > > > > > > > On 2026/01/21 03:52:29 Zhe Wang wrote: > > > > > > > Thanks for your response, now it looks good to me. > > > > > > > > > > > > > > Best regards, > > > > > > > Zhe Wang > > > > > > > > > > > > > > Liebing Yu <[email protected]> 于2026年1月20日周二 14:29写道： > > > > > > > > > > > > > > > Hi Zhe, sorry for the late reply. > > > > > > > > > > > > > > > > The primary focus of this FIP is not to address read/write > issues > > > > at > > > > > > the > > > > > > > > table or partition level, but rather to overcome limitations > at > > > the > > > > > > cluster > > > > > > > > level. Given the current capabilities of object storage, > > > read/write > > > > > > > > performance for a single table or partition is unlikely to > be a > > > > > > bottleneck; > > > > > > > > however, for a large-scale Fluss cluster, it can easily > become > > > one. > > > > > > > > Therefore, the core objective here is to distribute the > > > > cluster-wide > > > > > > > > read/write traffic across multiple remote storage systems. > > > > > > > > > > > > > > > > Best regards, > > > > > > > > Liebing Yu > > > > > > > > > > > > > > > > > > > > > > > > On Wed, 14 Jan 2026 at 16:07, Zhe Wang < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Liebing, Thanks for the clarification. > > > > > > > > > >1. To clarify, the data is currently split by partition > level > > > > for > > > > > > > > > partitioned tables and by table for non-partitioned tables. > > > > > > > > > > > > > > > > > > Therefore the main aim of this FIP is improving the speed > of > > > read > > > > > > data > > > > > > > > from > > > > > > > > > different partitions, store data speed may still limit for > a > > > > single > > > > > > > > system? > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > Zhe Wang > > > > > > > > > > > > > > > > > > Liebing Yu <[email protected]> 于2026年1月13日周二 19:11写道： > > > > > > > > > > > > > > > > > > > Hi Zhe, Thanks for the questions! > > > > > > > > > > > > > > > > > > > > 1. To clarify, the data is currently split by partition > level > > > > for > > > > > > > > > > partitioned tables and by table for non-partitioned > tables. > > > > > > > > > > > > > > > > > > > > 2. Regarding RemoteStorageCleaner, you are absolutely > right. > > > > > > Supporting > > > > > > > > > > remote.data.dirs there is necessary for a complete > cleanup > > > > when a > > > > > > table > > > > > > > > > is > > > > > > > > > > dropped. > > > > > > > > > > > > > > > > > > > > Thanks for pointing that out! > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > Liebing Yu > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, 12 Jan 2026 at 17:02, Zhe Wang < > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi Liebing, > > > > > > > > > > > > > > > > > > > > > > Thanks for driving this, I think it's a really useful > > > > feature. > > > > > > > > > > > I have two small questions: > > > > > > > > > > > 1. What's the scope for split data in dirs, I see > there's a > > > > > > > > partitionId > > > > > > > > > > in > > > > > > > > > > > ZK Data, so the data will spit by partition in > different > > > > > > directories, > > > > > > > > > or > > > > > > > > > > by > > > > > > > > > > > bucket? > > > > > > > > > > > 2. Maybe it needs to support remote.data.dirs in > > > > > > > > RemoteStorageCleaner? > > > > > > > > > So > > > > > > > > > > > we can delete all remoteStorage when delete table. > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > Zhe Wang > > > > > > > > > > > > > > > > > > > > > > Liebing Yu <[email protected]> 于2026年1月8日周四 20:10写道： > > > > > > > > > > > > > > > > > > > > > > > Hi devs, > > > > > > > > > > > > > > > > > > > > > > > > I propose initiating discussion on FIP-25[1]. Fluss > > > > leverages > > > > > > > > remote > > > > > > > > > > > > storage systems—such as Amazon S3, HDFS, and Alibaba > > > Cloud > > > > > > OSS—to > > > > > > > > > > > deliver a > > > > > > > > > > > > cost-efficient, highly available, and fault-tolerant > > > > storage > > > > > > > > solution > > > > > > > > > > > > compared to local disk. *However, in production > > > > environments, > > > > > > we > > > > > > > > > often > > > > > > > > > > > find > > > > > > > > > > > > that the bandwidth of a single remote storage > becomes a > > > > > > bottleneck. > > > > > > > > > > > *Taking > > > > > > > > > > > > OSS[2] as an example, the typical upload bandwidth > limit > > > > for > > > > > a > > > > > > > > single > > > > > > > > > > > > account is 20 Gbit/s (Internal) and 10 Gbit/s > (Public). > > > So > > > > I > > > > > > > > > initiated > > > > > > > > > > > this > > > > > > > > > > > > FIP which aims to introduce support for multiple > remote > > > > > storage > > > > > > > > paths > > > > > > > > > > and > > > > > > > > > > > > enables the dynamic addition of new storage paths > without > > > > > > service > > > > > > > > > > > > interruption. > > > > > > > > > > > > > > > > > > > > > > > > Any feedback and suggestions on this proposal are > > > welcome! > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLUSS/FIP-25%3A+Support+Multi-Location+for+Remote+Storage > > > > > > > > > > > > [2] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://www.alibabacloud.com/help/en/oss/user-guide/limits?spm=a2c63.l28256.help-menu-31815.d_0_0_5.2ac34d06oZYFvK > > > > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > Liebing Yu > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Lorenzo Affetti > > > > Senior Software Engineer @ Flink Team > > > > Ververica <http://www.ververica.com> > > > > > > > >
