That's an interesting idea. What do you mean by its own project? A couple of possibilities: - Spinning off a new ASF project - A separate Beam-governed repository (e.g. apache/beam-filesystems) - More clearly separate it in the current build system and release artifacts that allow it to be used independently
Personally I'd be resistant to the first two (I am a Google engineer and I like monorepos after all), but I don't see a major problem with the last one, except that it gives us another surface to maintain. Brian On Wed, May 19, 2021 at 8:38 PM Chad Dombrova <chad...@gmail.com> wrote: > This is a random idea, but the whole file IO system inside Beam would > actually be awesome to extract into its own project. IIRC, it’s not > particularly tied to Beam. > > I’m not saying this should be done now, but it’s be nice to keep it mind > for a future goal. > > -chad > > > > On Wed, May 19, 2021 at 10:23 AM Pablo Estrada <pabl...@google.com> wrote: > >> That would be great to add, Matt. Of course it's important to make this >> backwards compatible, but other than that, the addition would be very >> welcome. >> >> On Wed, May 19, 2021 at 9:41 AM Matt Rudary <matt.rud...@twosigma.com> >> wrote: >> >>> Hi, >>> >>> >>> >>> This is a quick sketch of a proposal – I wanted to get a sense of >>> whether there’s general support for this idea before fleshing it out >>> further, getting internal approvals, etc. >>> >>> >>> >>> I’m working with multiple storage systems that speak the S3 api. I would >>> like to support FileIO operations for these storage systems, but >>> S3FileSystem hardcodes the s3 scheme (the various systems use different URI >>> schemes) and it is in any case impossible to instantiate more than one in >>> the current design. >>> >>> >>> >>> I’d like to refactor the code in org.apache.beam.sdk.io.aws.s3 (and >>> maybe …aws.options) somewhat to enable this use-case. I haven’t worked out >>> the details yet, but it will take some thought to make this work in a >>> non-hacky way. >>> >>> >>> >>> Thanks >>> >>> Matt Rudary >>> >>
