Hi JB, Thanks a lot for your response. Does it mean all file-based IO will be added as schemes using IOChannelFactory (or the new name FileSystem). All others, e.g., HTTP, TCP, KV-store, DB, message-queue, should be source/sink transforms?
Thanks, Shen On Tue, Apr 11, 2017 at 10:29 AM, Jean-Baptiste Onofré <[email protected]> wrote: > Hi Shen, > > We are doing a refactoring of the file IO (IOChannelFactory). Thanks to > this refactoring, you will be able to use a scheme for hdfs (or s3, ...) > with different format (avro, text, hadoop input format, ...). > > It means that HdfsIO will be deprecated (to be removed at some point). I'm > working on couple of PRs to leverage the new file IO layer. > > Regards > JB > > > On 04/11/2017 03:56 PM, Shen Li wrote: > >> Hi, >> >> Is there any reason why HDFS IO is implemented as a BoundedSource while >> Google Cloud Storage is implemented as a scheme ("gs://") for TextIO? To >> contribute a new IO connector, how can I determine whether it should be >> implemented as a source transform or as a scheme for the TextIO? >> >> Thanks, >> >> Shen >> >> > -- > Jean-Baptiste Onofré > [email protected] > http://blog.nanthrax.net > Talend - http://www.talend.com > >
