+1 On Fri, Jun 2, 2017 at 12:06 PM, Bhupesh Chawda <bhup...@datatorrent.com> wrote:
> Hi, > > > > Emitting file > information > for a file based source like a file input operator > in malhar > seems > like > a > good > feature to provide. It is useful information for any downstream operator to > know > that a data tuple belongs to a certain file > for instance > . > > > We propose to add capability in the abstract file input operator to emit > file control tuples. These control tuples can include filenames as well as > any metadata that the user wishes to include along with it. > > To link this meta data to each tuple, we can add another port to the input > operator which would carry the meta data along with the actual tuple. We > can try to reduce the amount of meta data that goes with each tuple by > having some sort of meta encoding in the control tuple. > > > ~ Bhupesh > > > _______________________________________________________ > > Bhupesh Chawda > > E: bhup...@datatorrent.com | Twitter: @bhupeshsc > > www.datatorrent.com | apex.apache.org >
