Thanks! Shen
On Tue, Apr 11, 2017 at 11:10 AM, Jean-Baptiste Onofré <[email protected]> wrote: > Yes, FileSystem "plugins" will use a scheme. Other connectors will use (as > it's already the case) DoFn/Source transforms. > > Regards > JB > > > On 04/11/2017 05:05 PM, Shen Li wrote: > >> Hi JB, >> >> Thanks a lot for your response. Does it mean all file-based IO will be >> added as schemes using IOChannelFactory (or the new name FileSystem). All >> others, e.g., HTTP, TCP, KV-store, DB, message-queue, should be >> source/sink >> transforms? >> >> Thanks, >> >> Shen >> >> On Tue, Apr 11, 2017 at 10:29 AM, Jean-Baptiste Onofré <[email protected]> >> wrote: >> >> Hi Shen, >>> >>> We are doing a refactoring of the file IO (IOChannelFactory). Thanks to >>> this refactoring, you will be able to use a scheme for hdfs (or s3, ...) >>> with different format (avro, text, hadoop input format, ...). >>> >>> It means that HdfsIO will be deprecated (to be removed at some point). >>> I'm >>> working on couple of PRs to leverage the new file IO layer. >>> >>> Regards >>> JB >>> >>> >>> On 04/11/2017 03:56 PM, Shen Li wrote: >>> >>> Hi, >>>> >>>> Is there any reason why HDFS IO is implemented as a BoundedSource while >>>> Google Cloud Storage is implemented as a scheme ("gs://") for TextIO? To >>>> contribute a new IO connector, how can I determine whether it should be >>>> implemented as a source transform or as a scheme for the TextIO? >>>> >>>> Thanks, >>>> >>>> Shen >>>> >>>> >>>> -- >>> Jean-Baptiste Onofré >>> [email protected] >>> http://blog.nanthrax.net >>> Talend - http://www.talend.com >>> >>> >>> >> > -- > Jean-Baptiste Onofré > [email protected] > http://blog.nanthrax.net > Talend - http://www.talend.com >
