dennishuo commented on issue #156: URL: https://github.com/apache/polaris/issues/156#issuecomment-2309041176
Got a bit delayed from working on this, but I think I've at least gotten to a clear picture of how we might want to segregate scenarios for how callers can interact with `BasePolarisCatalog` to obtain a `FileIO` instance. Mostly just thinking out loud here for posterity - one problem is that since we wanted to leverage `BaseMetastoreCatalog` to avoid re-implementing all the `TableOperations` logic, there's potential for conflating things that are supposed to happen logically on the "client-side" vs on the "service-side". Basically we run the base Iceberg CatalogTests and View equivalents in two modes: - `PolarisRestCatalogIntegrationTest` - end-to-end logic that plumbs `CatalogTests` through a `RESTSessionCatalog` so that `BasePolarisCatalog` is only ever used for "server-side" actions (e.g. only needs to interact with metadata JSON files) - `BasePolarisCatalogTest` - directly points `CatalogTests` at the `BasePolarisCatalog` as if it's a traditional client-side-defined Iceberg catalog where "local" Transaction objects are used and `TableOperations::io()` is used for the FileIOs that potentially interact with manifest files and manifest-list files Roughly, there are three categories of tests to consider: 1. Tests that use the `BasePolarisCatalog` to obtain a "client-local" FileIO -- these should either manually create a FileIO for pre-creating data or can go through a config setting that forces `BasePolarisCatalog` to create a "defaultFileIO" using `catalogProperties`. 2. Tests which go through the full REST plumbing but need the "server-side" Catalog to still use special or mocked FileIO settings. For example, `PolarisSparkIntegrationTest` uses `S3MockContainer` and needs to be able to set `s3.endpoint` int he catalog config 3. Tests which go through the full REST plumbing and fully cooperate with "standard" credential-vending semantics and/or use only `file:///` paths. These should work without needing to fall back to `catalogProperties` nor rely on `defaultFileIO` Right now these rely on a lot of silent fallthrough behaviors, but we should make them more explicit. Supporting (2) more explicitly may also be the key to supporting https://github.com/apache/polaris/issues/69 in a cleaner way. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
