I don't think the lineage thing is even turned on in Tachyon - it was mostly a research prototype, so I don't think it'd make sense for us to use that.
On Thu, Dec 11, 2014 at 3:51 PM, Andrew Ash <[email protected]> wrote: > I'm interested in understanding this as well. One of the main ways Tachyon > is supposed to realize performance gains without sacrificing durability is > by storing the lineage of data rather than full copies of it (similar to > Spark). But if Spark isn't sending lineage information into Tachyon, then > I'm not sure how this isn't a durability concern. > > On Wed, Dec 10, 2014 at 5:47 AM, Jun Feng Liu <[email protected]> wrote: > > > Dose Spark today really leverage Tachyon linage to process data? It seems > > like the application should call createDependency function in TachyonFS > > to create a new linage node. But I did not find any place call that in > > Spark code. Did I missed anything? > > > > Best Regards > > > > > > *Jun Feng Liu* > > IBM China Systems & Technology Laboratory in Beijing > > > > ------------------------------ > > [image: 2D barcode - encoded with contact information] *Phone: > *86-10-82452683 > > > > * E-mail:* *[email protected]* <[email protected]> > > [image: IBM] > > > > BLD 28,ZGC Software Park > > No.8 Rd.Dong Bei Wang West, Dist.Haidian Beijing 100193 > > China > > > > > > > > > > >
