This is a good scope. And given there are multiple choices, an advanced student can expand scope to do both.
Kenn On Thu, Feb 21, 2019 at 5:36 PM Austin Bennett <whatwouldausti...@gmail.com> wrote: > Hi Pablo, > > Agree on the usefulness. > > Some thoughts embedded: > > > On Thu, Feb 21, 2019 at 3:19 PM Pablo Estrada <pabl...@google.com> wrote: > >> Hello all, >> I was thinking that a filesystem with support for s3 would be great to >> have in the Python SDK. If I am not wrong, it would simply involve >> implementing the filesystem classes >> <https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/filesystem.py> >> with >> s3, right? >> > > If talking about extending filesytems to alternate clouds -- would Azure > Blob storage also be sensible (not something I need, but imagine could be > valuable and pretty easy as I (naively) think these (gcs/azure-blob/s3 are > largely interchangeable). > > >> I am not familiar enough with s3, nor with filesystems, nor with AWS in >> general - but I have some outstanding questions: >> >> - Does this mean that we probably would need an extra [s3] target for >> installing apache_beam, like we do with [gcp]? >> - Not strictly necessary, but probably desirable... >> >> To think about: would there be [aws] to encompass more potential/future > aws things, to make it akin to gcp, or equivalently, would we want a > [gcs/gs] target to narrow down what gets loaded? > > >> - How do we handle KMS in GCS filesystem? >> - Would the filesystem encapsulation make KMS support in an s3 >> filesystem difficult? >> - Or even more... is the KMS support in AWS very different than in >> GCP? >> - I'd love comments from anyone informed around this : ) >> >> I use KMS with AWS, the tricky part is custom managed keys. I haven't > dug in enough to see how similar/different GCS implementation is (I thought > I only saw keys managed by GCP, so potentially easier, though AWS does have > that option). > > > >> >> - Is this project of an appropriate size for a GSoC student? >> >> Can't speak to appropriate size; was this listed as a project? Did we > have sufficiently vague proposals? I thought applications had been turned > in? > > > > >> Thoughts? >> Best >> -P. >> >
