I am not suggesting duplicating code; extend the operators. Just add something (may not even be a function) that can be viewed as specific to a particular source. Say for NFS, it may be as simple as changing a default. A file with NFS in its name help a great deal with adoption.
Thks Amol On Thu, May 5, 2016 at 11:45 AM, Chandni Singh <[email protected]> wrote: > IMO this is not a good idea. > > We are proposing to add additional Java code which is generic (works with > HDFS, NFS, local FS) but just calling it something specific - NFS. IMO this > is much more confusing to users. > > If we want to make it easier for users to find out that the FS Module > supports writing to NFS then maybe we need to improve documentation or > highlight it somewhere else. > > Adding java classes means more maintenance overhead and here these classes > are not doing anything additional. > > Thanks, > Chandni > > > > > > On Thu, May 5, 2016 at 11:24 AM, Mohit Jotwani <[email protected]> > wrote: > > > +1 on Sandeep's suggestion. This would make an end user's life lot more > > easier! > > > > Regards, > > Mohit > > > > On Thu, May 5, 2016 at 11:51 PM, Sandeep Deshmukh < > [email protected] > > > > > wrote: > > > > > I do agree with Amol on having clear and explicit modules. This is more > > > from an end user perspective. For someone who is new to Apex, having > > > separate NFS, HDFS, FTP, etc would make lot more sense than one generic > > FS > > > module. However small change these modules may have, like just couple > of > > > small functions, I would like to have them separate for the end user. > > > > > > It is finally about the perspective and the user experience :) > > > > > > Regards, > > > Sandeep > > > > > > On Thu, May 5, 2016 at 8:48 PM, Thomas Weise <[email protected]> > > > wrote: > > > > > > > I don't think we should name something NFS* when it isn't specific to > > > NFS. > > > > It is just like any other local FS for this purpose and that's > already > > > > covered by the Hadoop file system abstraction. > > > > > > > > Why can't a single FS Input module accommodate all of this. Once you > > know > > > > the FS URL, you can automatically optimize the configuration, if > > > > appropriate. > > > > > > > > Thanks, > > > > Thomas > > > > > > > > > > > > On Thu, May 5, 2016 at 12:08 AM, Chaitanya Chebolu < > > > > [email protected]> wrote: > > > > > > > > > Hi Chandni, > > > > > > > > > > Its a good point. I created the hierarchy based on user > perspective > > > and > > > > > especially for non Java users. If I return FileSplitter and > > BlockReader > > > > > from FS Input Module, then this module works for NFS. But, for > users > > > > > perspective it would be difficult, whether this module works for > NFS > > or > > > > any > > > > > other fileSystem. > > > > > > > > > > Regards, > > > > > Chaitanya > > > > > > > > > > On Thu, May 5, 2016 at 11:05 AM, Chandni Singh < > > > [email protected]> > > > > > wrote: > > > > > > > > > > > I am sorry Chaitanya but I have more questions about this > > > > > > > > > > > > 1. why is the FS Input Module abstract when by default it can > > return > > > > > > FileSplitter & BlockReader in com.datatorrent.lib.io.fs? > > > > > > These implementations are not specific to NFS. > > > > > > > > > > > > 2. In the NFS module that you have suggested to create, what is > > > > specific > > > > > to > > > > > > NFS? > > > > > > > > > > > > Please note: I have created a ticket APEXMALHAR-2081 to remove > > > > > > FSFileSplitter from library and move its feature to the base > > > operator. > > > > > > > > > > > > Thanks, > > > > > > Chandni > > > > > > > > > > > > On Wed, May 4, 2016 at 10:29 PM, Chaitanya Chebolu < > > > > > > [email protected]> wrote: > > > > > > > > > > > > > FSFileSplitter & BlockReader are available in > > > > com.datatorrent.lib.io.fs > > > > > > > package. > > > > > > > > > > > > > > On Thu, May 5, 2016 at 10:47 AM, Chandni Singh < > > > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > Ok. What is specific about the fileSplitter and blockReader > > > > returned > > > > > by > > > > > > > > this implementation? > > > > > > > > > > > > > > > > > > > > > > > > On May 4, 2016 9:43 PM, "Chaitanya Chebolu" < > > > > > [email protected] > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Chandni, > > > > > > > > > > > > > > > > > > Properties wise nothing specific. FS Input Module is an > > > abstract > > > > > > Module > > > > > > > > and > > > > > > > > > NFS Module implements the abstract methods - > > > createFileSplitter() > > > > > and > > > > > > > > > createBlockReader(). > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > Chaitanya > > > > > > > > > > > > > > > > > > On Wed, May 4, 2016 at 9:45 PM, Chandni Singh < > > > > > > [email protected] > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Chaitanya, > > > > > > > > > > > > > > > > > > > > What will be specific in NFS Input Module that is not > > > provided > > > > by > > > > > > FS > > > > > > > > > Input > > > > > > > > > > Module? > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Chandni > > > > > > > > > > > > > > > > > > > > On Wed, May 4, 2016 at 7:12 AM, Amol Kekre < > > > > [email protected] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > > > > > > > > > Thks > > > > > > > > > > > Amol > > > > > > > > > > > > > > > > > > > > > > On Tue, May 3, 2016 at 10:06 PM, Sandeep Deshmukh < > > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > Sandeep > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 29, 2016 at 3:26 PM, Mohit Jotwani < > > > > > > > > > [email protected]> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > Mohit > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 29, 2016 at 2:09 PM, Chaitanya Chebolu > < > > > > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > > > > > > > I am proposing NFS Input Module. Use case is to > > > read > > > > > > large > > > > > > > > > files > > > > > > > > > > > from > > > > > > > > > > > > > NFS > > > > > > > > > > > > > > in parallel. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Design of NFS input module: > > > > > > > > > > > > > > > > > > > > > > > > > > > > There is a common interface "FSInputModule" in > > > > Malhar > > > > > > for > > > > > > > > the > > > > > > > > > > > input > > > > > > > > > > > > > > Modules. NFS input Module extends from > > FSInputModule > > > > and > > > > > > can > > > > > > > be > > > > > > > > > > > > achieved > > > > > > > > > > > > > by > > > > > > > > > > > > > > using FSFileSplitter and BlockReader operators. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please share your thoughts on this. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > Chaitanya > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
