Python SDK doesn't have FileIO yet so let's keep ReadAllFromFoo transforms currently available for various file types around till we have that.
Thanks, Cham On Fri, Feb 1, 2019 at 7:41 AM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > Hi, > > readFiles() should be used IMHO. We should remove readAll() to avoid > confusion. > > Regards > JB > > On 30/01/2019 17:25, Ismaël Mejía wrote: > > Hello, > > > > A ‘recent’ pattern of use in Beam is to have in file based IOs a > > `readAll()` implementation that basically matches a `PCollection` of > > file patterns and reads them, e.g. `TextIO`, `AvroIO`. `ReadAll` is > > implemented by a expand function that matches files with FileIO and > > then reads them using a format specific `ReadFiles` transform e.g. > > TextIO.ReadFiles, AvroIO.ReadFiles. So in the end `ReadAll` in the > > Java implementation is just an user friendly API to hide FileIO.match > > + ReadFiles. > > > > Most recent IOs do NOT implement ReadAll to encourage the more > > composable approach of File + ReadFiles, e.g. XmlIO and ParquetIO. > > > > Implementing ReadAll as a wrapper is relatively easy and is definitely > > user friendly, but it has an issue, it may be error-prone and it adds > > more code to maintain (mostly ‘repeated’ code). However `readAll` is a > > more abstract pattern that applies not only to File based IOs so it > > makes sense for example in other transforms that map a `Pcollection` > > of read requests and is the basis for SDF composable style APIs like > > the recent `HBaseIO.readAll()`. > > > > So the question is should we: > > > > [1] Implement `readAll` in all file based IOs to be user friendly and > > assume the (minor) maintenance cost > > > > or > > > > [2] Deprecate `readAll` from file based IOs and encourage users to use > > FileIO + `readFiles` (less maintenance and encourage composition). > > > > I just checked quickly in the python code base but I did not find if > > the File match + ReadFiles pattern applies, but it would be nice to > > see what the python guys think on this too. > > > > This discussion comes from a recent slack conversation with Łukasz > > Gajowy, and we wanted to settle into one approach to make the IO > > signatures consistent, so any opinions/preferences? > > > > -- > Jean-Baptiste Onofré > jbono...@apache.org > http://blog.nanthrax.net > Talend - http://www.talend.com >
