This is a great question! I filed
https://issues.apache.org/jira/browse/BEAM-1929 to update the I/O docs to
make sure they answer this.
S
On Tue, Apr 11, 2017 at 8:20 AM Shen Li <cs.she...@gmail.com> wrote:
> Thanks!
>
> Shen
>
> On Tue, Apr 11, 2017 at 11:10 AM, Jean-Baptiste Onofré <j...@nanthrax.net>
> wrote:
>
> > Yes, FileSystem "plugins" will use a scheme. Other connectors will use
> (as
> > it's already the case) DoFn/Source transforms.
> >
> > Regards
> > JB
> >
> >
> > On 04/11/2017 05:05 PM, Shen Li wrote:
> >
> >> Hi JB,
> >>
> >> Thanks a lot for your response. Does it mean all file-based IO will be
> >> added as schemes using IOChannelFactory (or the new name FileSystem).
> All
> >> others, e.g., HTTP, TCP, KV-store, DB, message-queue, should be
> >> source/sink
> >> transforms?
> >>
> >> Thanks,
> >>
> >> Shen
> >>
> >> On Tue, Apr 11, 2017 at 10:29 AM, Jean-Baptiste Onofré <j...@nanthrax.net
> >
> >> wrote:
> >>
> >> Hi Shen,
> >>>
> >>> We are doing a refactoring of the file IO (IOChannelFactory). Thanks to
> >>> this refactoring, you will be able to use a scheme for hdfs (or s3,
> ...)
> >>> with different format (avro, text, hadoop input format, ...).
> >>>
> >>> It means that HdfsIO will be deprecated (to be removed at some point).
> >>> I'm
> >>> working on couple of PRs to leverage the new file IO layer.
> >>>
> >>> Regards
> >>> JB
> >>>
> >>>
> >>> On 04/11/2017 03:56 PM, Shen Li wrote:
> >>>
> >>> Hi,
> >>>>
> >>>> Is there any reason why HDFS IO is implemented as a BoundedSource
> while
> >>>> Google Cloud Storage is implemented as a scheme ("gs://") for TextIO?
> To
> >>>> contribute a new IO connector, how can I determine whether it should
> be
> >>>> implemented as a source transform or as a scheme for the TextIO?
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Shen
> >>>>
> >>>>
> >>>> --
> >>> Jean-Baptiste Onofré
> >>> jbono...@apache.org
> >>> http://blog.nanthrax.net
> >>> Talend - http://www.talend.com
> >>>
> >>>
> >>>
> >>
> > --
> > Jean-Baptiste Onofré
> > jbono...@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>
