Hi Antoine, > Our main concern is that this new arrow::dataset::RadosFormat class will > be > > deriving from the arrow::dataset::FileFormat class, which seems to raise > a > > conceptual mismatch as there isn’t really a RADOS format but rather a > > formatting/serialization deferral that will be taking place, effectively > > introducing a new client-server layer in the Dataset API. > > So, RadosFormat would ultimately redirect to another dataset format > (e.g. ParquetFormat) when it comes to actually understanding the data? >
Yes, that is our plan. Since this is going to be done on the storage-, server-side, this would be transparent to the client. So our main concern is whether this be OK from the design perspective, and could this eventually be merged upstream? thanks!
