prantogg commented on PR #71: URL: https://github.com/apache/sedona-spatialbench/pull/71#issuecomment-3893048841
> Thank you! > > I don't think it is necessary for this PR, but the `ObjectStore` may be a good future choice (here it would let you unify the local and s3 code path and make it easier to add future gcs or azure support). > > It may be worth setting up a test for this in CI using MinIO or another S3 alternative as otherwise I don't think there are any tests here to validate the behaviour. Thanks @paleolimbot! Agreed on `ObjectStore` for local path. The current implementation for local path is an exact copy of tpchgen-rs project. Happy to tackle that as a follow-up! On the MinIO CI test - my thinking was that since `S3Writer` implements `std::io::Write`, the datagen pipeline is agnostic to the destination. The unit tests validate the writer logic (buffering, multipart, finalization) against an InMemory object store, which covers our code. Testing against an actual S3 endpoint would mostly be validating `object_store::aws::AmazonS3Builder`'s backend rather than ours? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
