Hi, Support for multiple storage backends including remote storage backends is a pretty high value piece of functionality. I am happy to see there is interest in that.
I think that `ChannelProxyFactory` as an integration point is going to quickly turn into a dead end as we get into really using multiple storage backends. We need to be able to list files and really the full range of filesystem interactions that Java supports should work with any backend to make development, testing, and using existing code straightforward. It's a little more work to get C* to creates paths for alternate backends where appropriate, but that works is probably necessary even with `ChanelProxyFactory` and munging UNIX paths (vs supporting multiple Fileystems). There will probably also be backend specific behaviors that show up above the `ChannelProxy` layer that will depend on the backend. Ideally there would be some config to specify several backend filesystems and their individual configuration that can be used, as well as configuration and support for a "backend file router" for file creation (and opening) that can be used to route files to the backend most appropriate. Regards, Ariel On Mon, Sep 25, 2023, at 2:48 AM, Claude Warren, Jr via dev wrote: > I have just filed CEP-36 [1] to allow for keyspace/table storage outside of > the standard storage space. > > There are two desires driving this change: > 1. The ability to temporarily move some keyspaces/tables to storage outside > the normal directory tree to other disk so that compaction can occur in > situations where there is not enough disk space for compaction and the > processing to the moved data can not be suspended. > 2. The ability to store infrequently used data on slower cheaper storage > layers. > I have a working POC implementation [2] though there are some issues still to > be solved and much logging to be reduced. > > I look forward to productive discussions, > Claude > > [1] > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-36%3A+A+Configurable+ChannelProxy+to+alias+external+storage+locations > [2] https://github.com/Claudenw/cassandra/tree/channel_proxy_factory > >
