Re: Are tachyon and akka removed from 2.1.1 please

[??????????]

thanks gromakowski and chin wei.


 
---Original---
From: "vincent gromakowski"<vincent.gromakow...@gmail.com>
Date: 2017/5/23 00:54:33
To: "Chin Wei Low"<lowchin...@gmail.com>;
Cc: "user"<user@spark.apache.org>;"??"<1427357...@qq.com>;"Gene 
Pang"<gene.p...@gmail.com>;
Subject: Re: Are tachyon and akka removed from 2.1.1 please


Akka has been replaced by netty in 1.6

Le 22 mai 2017 15:25, "Chin Wei Low" <lowchin...@gmail.com> a ??crit :
I think akka has been removed since 2.0.

On 22 May 2017 10:19 pm, "Gene Pang" <gene.p...@gmail.com> wrote:
Hi,

Tachyon has been renamed to Alluxio. Here is the documentation for running 
Alluxio with Spark.


Hope this helps,
Gene


On Sun, May 21, 2017 at 6:15 PM, ?? <1427357...@qq.com> wrote:
HI all,
Iread some paper about source code, the paper base on version 1.2.  they refer 
the tachyon and akka.  When i read the 2.1code. I can not find the code abiut 
akka and tachyon.


Are tachyon and akka removed from 2.1.1  please

Re: Are tachyon and akka removed from 2.1.1 please

[??????????]

thanks Gene.


 
---Original---
From: "Gene Pang"<gene.p...@gmail.com>
Date: 2017/5/22 22:19:47
To: "??"<1427357...@qq.com>;
Cc: "user"<user@spark.apache.org>;
Subject: Re: Are tachyon and akka removed from 2.1.1 please


Hi,

Tachyon has been renamed to Alluxio. Here is the documentation for running 
Alluxio with Spark.


Hope this helps,
Gene


On Sun, May 21, 2017 at 6:15 PM, ?? <1427357...@qq.com> wrote:
HI all,
Iread some paper about source code, the paper base on version 1.2.  they refer 
the tachyon and akka.  When i read the 2.1code. I can not find the code abiut 
akka and tachyon.


Are tachyon and akka removed from 2.1.1  please

Re: Are tachyon and akka removed from 2.1.1 please

[vincent gromakowski]

Akka has been replaced by netty in 1.6

Le 22 mai 2017 15:25, "Chin Wei Low" <lowchin...@gmail.com> a écrit :

> I think akka has been removed since 2.0.
>
> On 22 May 2017 10:19 pm, "Gene Pang" <gene.p...@gmail.com> wrote:
>
>> Hi,
>>
>> Tachyon has been renamed to Alluxio. Here is the documentation for
>> running Alluxio with Spark
>> <http://www.alluxio.org/docs/master/en/Running-Spark-on-Alluxio.html>.
>>
>> Hope this helps,
>> Gene
>>
>> On Sun, May 21, 2017 at 6:15 PM, 萝卜丝炒饭 <1427357...@qq.com> wrote:
>>
>>> HI all,
>>> Iread some paper about source code, the paper base on version 1.2.  they
>>> refer the tachyon and akka.  When i read the 2.1code. I can not find the
>>> code abiut akka and tachyon.
>>>
>>> Are tachyon and akka removed from 2.1.1  please
>>>
>>
>>

Re: Are tachyon and akka removed from 2.1.1 please

[Chin Wei Low]

I think akka has been removed since 2.0.

On 22 May 2017 10:19 pm, "Gene Pang" <gene.p...@gmail.com> wrote:

> Hi,
>
> Tachyon has been renamed to Alluxio. Here is the documentation for
> running Alluxio with Spark
> <http://www.alluxio.org/docs/master/en/Running-Spark-on-Alluxio.html>.
>
> Hope this helps,
> Gene
>
> On Sun, May 21, 2017 at 6:15 PM, 萝卜丝炒饭 <1427357...@qq.com> wrote:
>
>> HI all,
>> Iread some paper about source code, the paper base on version 1.2.  they
>> refer the tachyon and akka.  When i read the 2.1code. I can not find the
>> code abiut akka and tachyon.
>>
>> Are tachyon and akka removed from 2.1.1  please
>>
>
>

Re: Are tachyon and akka removed from 2.1.1 please

[Gene Pang]

Hi,

Tachyon has been renamed to Alluxio. Here is the documentation for running
Alluxio with Spark
<http://www.alluxio.org/docs/master/en/Running-Spark-on-Alluxio.html>.

Hope this helps,
Gene

On Sun, May 21, 2017 at 6:15 PM, 萝卜丝炒饭 <1427357...@qq.com> wrote:

> HI all,
> Iread some paper about source code, the paper base on version 1.2.  they
> refer the tachyon and akka.  When i read the 2.1code. I can not find the
> code abiut akka and tachyon.
>
> Are tachyon and akka removed from 2.1.1  please
>

Are tachyon and akka removed from 2.1.1 please

[??????????]

HI all,
Iread some paper about source code, the paper base on version 1.2.  they refer 
the tachyon and akka.  When i read the 2.1code. I can not find the code abiut 
akka and tachyon.


Are tachyon and akka removed from 2.1.1  please

Re: off heap to alluxio/tachyon in Spark 2

[Bin Fan]

Hi,

If you are looking for how to run Spark on Alluxio (formerly Tachyon),
here is the documentation from Alluxio doc site:
http://www.alluxio.org/docs/master/en/Running-Spark-on-Alluxio.html
It still works for Spark 2.x.

Alluxio team also published articles on when and why running Spark (2.x)
with Alluxio may benefit performance:
http://www.alluxio.com/2016/08/effective-spark-rdds-with-alluxio/

- Bin


On Mon, Sep 19, 2016 at 7:56 AM, aka.fe2s <aka.f...@gmail.com> wrote:

> Hi folks,
>
> What has happened with Tachyon / Alluxio in Spark 2? Doc doesn't mention
> it no longer.
>
> --
> Oleksiy Dyagilev
>

Re: off heap to alluxio/tachyon in Spark 2

[Sean Owen]

It backed the "OFF_HEAP" storage level for RDDs. That's not quite the
same thing that off-heap Tungsten allocation refers to.

It's also worth pointing out that things like HDFS also can put data
into memory already.

On Mon, Sep 19, 2016 at 7:48 PM, Richard Catlin
<richard.m.cat...@gmail.com> wrote:
> Here is my understanding.
>
> Spark used Tachyon as an off-heap solution for RDDs.  In certain situations,
> it would alleviate Garbage Collection or the RDDs.
>
> Tungsten, Spark 2’s off-heap (columnar format) is much more efficient and
> used as the default.  Alluvio no longer makes sense for this use.
>
>
> You can still use Tachyon/Alluxio to bring your files into Memory, which is
> quicker for Spark to access than your DFS(HDFS or S3).
>
> Alluxio actually supports a “Tiered Filesystem”, and automatically brings
> the “hotter” files into the fastest storage (Memory, SSD).  You can
> configure it with Memory, SSD, and/or HDDs with the DFS as the persistent
> store, called under-filesystem.
>
> Hope this helps.
>
> Richard Catlin
>
> On Sep 19, 2016, at 7:56 AM, aka.fe2s <aka.f...@gmail.com> wrote:
>
> Hi folks,
>
> What has happened with Tachyon / Alluxio in Spark 2? Doc doesn't mention it
> no longer.
>
> --
> Oleksiy Dyagilev
>
>

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: off heap to alluxio/tachyon in Spark 2

[Richard Catlin]

Here is my understanding.

Spark used Tachyon as an off-heap solution for RDDs.  In certain situations, it 
would alleviate Garbage Collection or the RDDs.

Tungsten, Spark 2’s off-heap (columnar format) is much more efficient and used 
as the default.  Alluvio no longer makes sense for this use.


You can still use Tachyon/Alluxio to bring your files into Memory, which is 
quicker for Spark to access than your DFS(HDFS or S3).

Alluxio actually supports a “Tiered Filesystem”, and automatically brings the 
“hotter” files into the fastest storage (Memory, SSD).  You can configure it 
with Memory, SSD, and/or HDDs with the DFS as the persistent store, called 
under-filesystem.

Hope this helps.

Richard Catlin

> On Sep 19, 2016, at 7:56 AM, aka.fe2s <aka.f...@gmail.com> wrote:
> 
> Hi folks,
> 
> What has happened with Tachyon / Alluxio in Spark 2? Doc doesn't mention it 
> no longer.
> 
> --
> Oleksiy Dyagilev

off heap to alluxio/tachyon in Spark 2

[aka.fe2s]

Hi folks,

What has happened with Tachyon / Alluxio in Spark 2? Doc doesn't mention it
no longer.

--
Oleksiy Dyagilev

Re: TTransportException when using Spark 1.6.0 on top of Tachyon 0.8.2

[Jia Zou]

Hi, Calvin, I am running  24GB data Spark KMeans in a c3.2xlarge AWS 
instance with 30GB physical memory.
Spark will cache data off-heap to Tachyon, the input data is also stored in 
Tachyon.
Tachyon is configured to use 15GB memory, and use tired store.
Tachyon underFS is /tmp.

The only configuration I've changed is Tachyon data block size.

Above experiment is a part of a research project.

Best Regards,
Jia

On Thursday, January 28, 2016 at 9:11:19 PM UTC-6, Calvin Jia wrote:
>
> Hi,
>
> Thanks for the detailed information. How large is the dataset you are 
> running against? Also did you change any Tachyon configurations?
>
> Thanks,
> Calvin
>

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: TTransportException when using Spark 1.6.0 on top of Tachyon 0.8.2

[cc]

Hey, Jia Zou

I'm curious about this exception, the error log you showed that the 
exception is related to unlockBlock, could you upload your full master.log 
and worker.log under tachyon/logs directory? 

Best,
Cheng

在 2016年1月29日星期五 UTC+8上午11:11:19，Calvin Jia写道：
>
> Hi,
>
> Thanks for the detailed information. How large is the dataset you are 
> running against? Also did you change any Tachyon configurations?
>
> Thanks,
> Calvin
>

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: TTransportException when using Spark 1.6.0 on top of Tachyon 0.8.2

[Calvin Jia]

Hi,

Thanks for the detailed information. How large is the dataset you are 
running against? Also did you change any Tachyon configurations?

Thanks,
Calvin

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

TTransportException when using Spark 1.6.0 on top of Tachyon 0.8.2

[Jia Zou]

Dears, I keep getting below exception when using Spark 1.6.0 on top of
Tachyon 0.8.2. Tachyon is 93% used and configured as CACHE_THROUGH.

Any suggestions will be appreciated, thanks!

=

Exception in thread "main" org.apache.spark.SparkException: Job aborted due
to stage failure: Task 13 in stage 0.0 failed 4 times, most recent failure:
Lost task 13.3 in stage 0.0 (TID 33, ip-10-73-198-35.ec2.internal):
java.io.IOException: tachyon.org.apache.thrift.transport.TTransportException

at tachyon.worker.WorkerClient.unlockBlock(WorkerClient.java:416)

at tachyon.client.block.LocalBlockInStream.close(LocalBlockInStream.java:87)

at tachyon.client.file.FileInStream.close(FileInStream.java:105)

at tachyon.hadoop.HdfsFileInputStream.read(HdfsFileInputStream.java:171)

at java.io.DataInputStream.readInt(DataInputStream.java:388)

at
org.apache.hadoop.io.SequenceFile$Reader.readRecordLength(SequenceFile.java:2325)

at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2356)

at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2493)

at
org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:82)

at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:246)

at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:208)

at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)

at
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)

at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:369)

at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:193)

at
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)

at
org.apache.spark.rdd.RDD$$anonfun$zip$1$$anonfun$apply$31$$anon$1.hasNext(RDD.scala:851)

at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:369)

at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1595)

at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1143)

at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1143)

at
org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)

at
org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)

at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)

at org.apache.spark.scheduler.Task.run(Task.scala:89)

at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)

at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Caused by: tachyon.org.apache.thrift.transport.TTransportException

at
tachyon.org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)

at
tachyon.org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)

at
tachyon.org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)

at
tachyon.org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)

at
tachyon.org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)

at
tachyon.org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)

at
tachyon.org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)

at
tachyon.org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)

at
tachyon.org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)

at
tachyon.thrift.WorkerService$Client.recv_unlockBlock(WorkerService.java:455)

at tachyon.thrift.WorkerService$Client.unlockBlock(WorkerService.java:441)

at tachyon.worker.WorkerClient.unlockBlock(WorkerClient.java:413)

... 28 more

Re: TTransportException when using Spark 1.6.0 on top of Tachyon 0.8.2

[Jia Zou]

BTW. The tachyon worker log says following:



2015-12-27 01:33:44,599 ERROR WORKER_LOGGER
(WorkerBlockMasterClient.java:getId) - java.net.SocketException: Connection
reset

org.apache.thrift.transport.TTransportException: java.net.SocketException:
Connection reset

at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)

at
org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)

at
org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)

at
org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)

at
org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)

at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)

at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)

at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)

at
org.apache.thrift.protocol.TProtocolDecorator.readMessageBegin(TProtocolDecorator.java:135)

at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)

at
tachyon.thrift.BlockMasterService$Client.recv_workerGetWorkerId(BlockMasterService.java:235)

at
tachyon.thrift.BlockMasterService$Client.workerGetWorkerId(BlockMasterService.java:222)

at
tachyon.client.WorkerBlockMasterClient.getId(WorkerBlockMasterClient.java:103)

at
tachyon.worker.WorkerIdRegistry.registerWithBlockMaster(WorkerIdRegistry.java:59)

at tachyon.worker.block.BlockWorker.(BlockWorker.java:200)

at tachyon.worker.TachyonWorker.main(TachyonWorker.java:42)

Caused by: java.net.SocketException: Connection reset

at java.net.SocketInputStream.read(SocketInputStream.java:196)

at java.net.SocketInputStream.read(SocketInputStream.java:122)

at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)

at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)

at java.io.BufferedInputStream.read(BufferedInputStream.java:334)

at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)

... 15 more

On Wed, Jan 27, 2016 at 5:02 AM, Jia Zou <jacqueline...@gmail.com> wrote:

> Dears, I keep getting below exception when using Spark 1.6.0 on top of
> Tachyon 0.8.2. Tachyon is 93% used and configured as CACHE_THROUGH.
>
> Any suggestions will be appreciated, thanks!
>
> =
>
> Exception in thread "main" org.apache.spark.SparkException: Job aborted
> due to stage failure: Task 13 in stage 0.0 failed 4 times, most recent
> failure: Lost task 13.3 in stage 0.0 (TID 33,
> ip-10-73-198-35.ec2.internal): java.io.IOException:
> tachyon.org.apache.thrift.transport.TTransportException
>
> at tachyon.worker.WorkerClient.unlockBlock(WorkerClient.java:416)
>
> at
> tachyon.client.block.LocalBlockInStream.close(LocalBlockInStream.java:87)
>
> at tachyon.client.file.FileInStream.close(FileInStream.java:105)
>
> at tachyon.hadoop.HdfsFileInputStream.read(HdfsFileInputStream.java:171)
>
> at java.io.DataInputStream.readInt(DataInputStream.java:388)
>
> at
> org.apache.hadoop.io.SequenceFile$Reader.readRecordLength(SequenceFile.java:2325)
>
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2356)
>
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2493)
>
> at
> org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:82)
>
> at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:246)
>
> at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:208)
>
> at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
>
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:369)
>
> at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:193)
>
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>
> at
> org.apache.spark.rdd.RDD$$anonfun$zip$1$$anonfun$apply$31$$anon$1.hasNext(RDD.scala:851)
>
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:369)
>
> at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1595)
>
> at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1143)
>
> at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1143)
>
> at
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
>
> at
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
>
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
&g

Re: TTransportException when using Spark 1.6.0 on top of Tachyon 0.8.2

[Jia Zou]

BTW. At the end of the log, I also find a lot of errors like below:

=

2016-01-27 11:47:18,515 ERROR server.TThreadPoolServer
(TThreadPoolServer.java:run) - Error occurred during processing of message.

java.lang.NullPointerException

at
tachyon.worker.block.BlockLockManager.unlockBlock(BlockLockManager.java:142)

at
tachyon.worker.block.TieredBlockStore.unlockBlock(TieredBlockStore.java:148)

at
tachyon.worker.block.BlockDataManager.unlockBlock(BlockDataManager.java:476)

at
tachyon.worker.block.BlockServiceHandler.unlockBlock(BlockServiceHandler.java:232)

at
tachyon.thrift.WorkerService$Processor$unlockBlock.getResult(WorkerService.java:1150)

at
tachyon.thrift.WorkerService$Processor$unlockBlock.getResult(WorkerService.java:1135)

at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)

at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)

at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)

at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)


On Wed, Jan 27, 2016 at 5:53 AM, Jia Zou <jacqueline...@gmail.com> wrote:

> BTW. The tachyon worker log says following:
>
> 
>
> 2015-12-27 01:33:44,599 ERROR WORKER_LOGGER
> (WorkerBlockMasterClient.java:getId) - java.net.SocketException: Connection
> reset
>
> org.apache.thrift.transport.TTransportException: java.net.SocketException:
> Connection reset
>
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
>
> at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
>
> at
> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
>
> at
> org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
>
> at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
>
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
>
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
>
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
>
> at
> org.apache.thrift.protocol.TProtocolDecorator.readMessageBegin(TProtocolDecorator.java:135)
>
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
>
> at
> tachyon.thrift.BlockMasterService$Client.recv_workerGetWorkerId(BlockMasterService.java:235)
>
> at
> tachyon.thrift.BlockMasterService$Client.workerGetWorkerId(BlockMasterService.java:222)
>
> at
> tachyon.client.WorkerBlockMasterClient.getId(WorkerBlockMasterClient.java:103)
>
> at
> tachyon.worker.WorkerIdRegistry.registerWithBlockMaster(WorkerIdRegistry.java:59)
>
> at tachyon.worker.block.BlockWorker.(BlockWorker.java:200)
>
> at tachyon.worker.TachyonWorker.main(TachyonWorker.java:42)
>
> Caused by: java.net.SocketException: Connection reset
>
> at java.net.SocketInputStream.read(SocketInputStream.java:196)
>
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
>
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
>
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>
> ... 15 more
>
> On Wed, Jan 27, 2016 at 5:02 AM, Jia Zou <jacqueline...@gmail.com> wrote:
>
>> Dears, I keep getting below exception when using Spark 1.6.0 on top of
>> Tachyon 0.8.2. Tachyon is 93% used and configured as CACHE_THROUGH.
>>
>> Any suggestions will be appreciated, thanks!
>>
>> =
>>
>> Exception in thread "main" org.apache.spark.SparkException: Job aborted
>> due to stage failure: Task 13 in stage 0.0 failed 4 times, most recent
>> failure: Lost task 13.3 in stage 0.0 (TID 33,
>> ip-10-73-198-35.ec2.internal): java.io.IOException:
>> tachyon.org.apache.thrift.transport.TTransportException
>>
>> at tachyon.worker.WorkerClient.unlockBlock(WorkerClient.java:416)
>>
>> at
>> tachyon.client.block.LocalBlockInStream.close(LocalBlockInStream.java:87)
>>
>> at tachyon.client.

Re: TTransportException when using Spark 1.6.0 on top of Tachyon 0.8.2

[Jia Zou]

BTW. the error happens when configure Spark to read input file from Tachyon
like following:

/home/ubuntu/spark-1.6.0/bin/spark-submit  --properties-file
/home/ubuntu/HiBench/report/kmeans/spark/java/conf/sparkbench/spark.conf
--class org.apache.spark.examples.mllib.JavaKMeans --master spark://ip
-10-73-198-35:7077
/home/ubuntu/HiBench/src/sparkbench/target/sparkbench-5.0-SNAPSHOT-MR2-spark1.5-jar-with-dependencies.jar
tachyon://localhost:19998/Kmeans/Input/samples 10 5

On Wed, Jan 27, 2016 at 5:02 AM, Jia Zou <jacqueline...@gmail.com> wrote:

> Dears, I keep getting below exception when using Spark 1.6.0 on top of
> Tachyon 0.8.2. Tachyon is 93% used and configured as CACHE_THROUGH.
>
> Any suggestions will be appreciated, thanks!
>
> =
>
> Exception in thread "main" org.apache.spark.SparkException: Job aborted
> due to stage failure: Task 13 in stage 0.0 failed 4 times, most recent
> failure: Lost task 13.3 in stage 0.0 (TID 33,
> ip-10-73-198-35.ec2.internal): java.io.IOException:
> tachyon.org.apache.thrift.transport.TTransportException
>
> at tachyon.worker.WorkerClient.unlockBlock(WorkerClient.java:416)
>
> at
> tachyon.client.block.LocalBlockInStream.close(LocalBlockInStream.java:87)
>
> at tachyon.client.file.FileInStream.close(FileInStream.java:105)
>
> at tachyon.hadoop.HdfsFileInputStream.read(HdfsFileInputStream.java:171)
>
> at java.io.DataInputStream.readInt(DataInputStream.java:388)
>
> at
> org.apache.hadoop.io.SequenceFile$Reader.readRecordLength(SequenceFile.java:2325)
>
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2356)
>
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2493)
>
> at
> org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:82)
>
> at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:246)
>
> at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:208)
>
> at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
>
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:369)
>
> at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:193)
>
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>
> at
> org.apache.spark.rdd.RDD$$anonfun$zip$1$$anonfun$apply$31$$anon$1.hasNext(RDD.scala:851)
>
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:369)
>
> at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1595)
>
> at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1143)
>
> at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1143)
>
> at
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
>
> at
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
>
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
>
> at org.apache.spark.scheduler.Task.run(Task.scala:89)
>
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: tachyon.org.apache.thrift.transport.TTransportException
>
> at
> tachyon.org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>
> at
> tachyon.org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
>
> at
> tachyon.org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
>
> at
> tachyon.org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
>
> at
> tachyon.org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
>
> at
> tachyon.org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
>
> at
> tachyon.org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
>
> at
> tachyon.org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
>
> at
> tachyon.org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
>
> at
> tachyon.thrift.WorkerService$Client.recv_unlockBlock(WorkerService.java:455)
>
> at tachyon.thrift.WorkerService$Client.unlockBlock(WorkerService.java:441)
>
> at tachyon.worker.WorkerClient.unlockBlock(WorkerClient.java:413)
>
> ... 28 more
>
>
>

Re: How to query data in tachyon with spark-sql

[Gene Pang]

Hi,

You should be able to point Hive to Tachyon instead of HDFS, and that
should allow Hive to access data in Tachyon. If Spark SQL was pointing to
an HDFS file, you could instead point it to a Tachyon file, and that should
work too.

Hope that helps,
Gene

On Wed, Jan 20, 2016 at 2:06 AM, Sea <261810...@qq.com> wrote:

> Hi,all
>  I want to mount some hive table in tachyon, but I don't know how to
> query data in tachyon with spark-sql, who knows?
>

How to query data in tachyon with spark-sql

[Sea]

Hi,all  I want to mount some hive table in tachyon, but I don't know how to 
query data in tachyon with spark-sql, who knows?

Re: Saving RDDs in Tachyon

[Calvin Jia]

Hi Mark,

Were you able to successfully store the RDD with Akhil's method? When you
read it back as an objectFile, you will also need to specify the correct
type.

You can find more information about integrating Spark and Tachyon on this
page: http://tachyon-project.org/documentation/Running-Spark-on-Tachyon.html
.

Hope this helps,
Calvin

On Fri, Oct 30, 2015 at 7:04 AM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:

> I guess you can do a .saveAsObjectFiles and read it back as sc.objectFile
>
> Thanks
> Best Regards
>
> On Fri, Oct 23, 2015 at 7:57 AM, mark <manwoodv...@googlemail.com> wrote:
>
>> I have Avro records stored in Parquet files in HDFS. I want to read these
>> out as an RDD and save that RDD in Tachyon for any spark job that wants the
>> data.
>>
>> How do I save the RDD in Tachyon? What format do I use? Which RDD
>> 'saveAs...' method do I want?
>>
>> Thanks
>>
>
>

how often you use Tachyon to accelerate Spark

[Arvin]

Hi,all
 Well, I have some questions about Tachyon and Spark. I found the
interactive between Spark and Tachyon is the caching RDD use off-heap. I
wonder if you guys use Tachyon frequently, such caching RDD by Tachyon? Is
this action(caching rdd by tachyon) has a profound effect to accelerate
Spark? 



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/how-often-you-use-Tachyon-to-accelerate-Spark-tp25613.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Saving RDDs in Tachyon

[Akhil Das]

I guess you can do a .saveAsObjectFiles and read it back as sc.objectFile

Thanks
Best Regards

On Fri, Oct 23, 2015 at 7:57 AM, mark <manwoodv...@googlemail.com> wrote:

> I have Avro records stored in Parquet files in HDFS. I want to read these
> out as an RDD and save that RDD in Tachyon for any spark job that wants the
> data.
>
> How do I save the RDD in Tachyon? What format do I use? Which RDD
> 'saveAs...' method do I want?
>
> Thanks
>

Re: How does Spark coordinate with Tachyon wrt data locality

[Calvin Jia]

Hi Shane,

Tachyon provides an api to get the block locations of the file which Spark
uses when scheduling tasks.

Hope this helps,
Calvin

On Fri, Oct 23, 2015 at 8:15 AM, Kinsella, Shane <shane.kinse...@aspect.com>
wrote:

> Hi all,
>
>
>
> I am looking into how Spark handles data locality wrt Tachyon. My main
> concern is how this is coordinated. Will it send a task based on a file
> loaded from Tachyon to a node that it knows has that file locally and how
> does it know which nodes has what?
>
>
>
> Kind regards,
>
> Shane
> This email (including any attachments) is proprietary to Aspect Software,
> Inc. and may contain information that is confidential. If you have received
> this message in error, please do not read, copy or forward this message.
> Please notify the sender immediately, delete it from your system and
> destroy any copies. You may not further disclose or distribute this email
> or its attachments.
>

How does Spark coordinate with Tachyon wrt data locality

[Kinsella, Shane]

Hi all,

I am looking into how Spark handles data locality wrt Tachyon. My main concern 
is how this is coordinated. Will it send a task based on a file loaded from 
Tachyon to a node that it knows has that file locally and how does it know 
which nodes has what?

Kind regards,
Shane
This email (including any attachments) is proprietary to Aspect Software, Inc. 
and may contain information that is confidential. If you have received this 
message in error, please do not read, copy or forward this message. Please 
notify the sender immediately, delete it from your system and destroy any 
copies. You may not further disclose or distribute this email or its 
attachments.

Saving RDDs in Tachyon

[mark]

I have Avro records stored in Parquet files in HDFS. I want to read these
out as an RDD and save that RDD in Tachyon for any spark job that wants the
data.

How do I save the RDD in Tachyon? What format do I use? Which RDD
'saveAs...' method do I want?

Thanks

Re: Spark Streaming with Tachyon : Data Loss on Receiver Failure due to WAL error

[N B]

Hi Dibyendu,

I am not sure I understand completely. But are you suggesting that
currently there is no way to enable Checkpoint directory to be in Tachyon?

Thanks
Nikunj


On Fri, Sep 25, 2015 at 11:49 PM, Dibyendu Bhattacharya <
dibyendu.bhattach...@gmail.com> wrote:

> Hi,
>
> Recently I was working on a PR to use Tachyon as OFF_HEAP store for Spark
> Streaming and make sure Spark Streaming can recover from Driver failure and
> recover the blocks form Tachyon.
>
> The The Motivation for this PR is  :
>
> If Streaming application stores the blocks OFF_HEAP, it may not need any
> WAL like feature to recover from Driver failure. As long as the writing of
> blocks to Tachyon from Streaming receiver is durable, it should be
> recoverable from Tachyon directly on Driver failure.
> This can solve the issue of expensive WAL write and duplicating the blocks
> both in MEMORY and also WAL and also guarantee end to end No-Data-Loss
> channel using OFF_HEAP store.
>
> https://github.com/apache/spark/pull/8817
>
> This PR still under review . But having done various fail over testing in
> my environment , I see this PR worked perfectly fine without any data loss
> . Let see what TD and other have to say on this PR .
>
> Below is the configuration I used to test this PR ..
>
>
> Spark : 1.6 from Master
> Tachyon : 0.7.1
>
> SparkConfiguration Details :
>
> SparkConf conf = new SparkConf().setAppName("TestTachyon")
> .set("spark.streaming.unpersist", "true")
> .set("spark.local.dir", "/mnt1/spark/tincan")
> .set("tachyon.zookeeper.address","10.252.5.113:2182")
> .set("tachyon.usezookeeper","true")
> .set("spark.externalBlockStore.url", "tachyon-ft://
> ip-10-252-5-113.asskickery.us:19998")
> .set("spark.externalBlockStore.baseDir", "/sparkstreaming")
> .set("spark.externalBlockStore.folderName","pearson")
> .set("spark.externalBlockStore.dirId", "subpub")
>
> .set("spark.externalBlockStore.keepExternalBlockStoreDirOnShutdown","true");
>
> JavaStreamingContext jsc = new JavaStreamingContext(conf, new Duration(
> 1));
>
> String checkpointDirectory = "hdfs://
> 10.252.5.113:9000/user/hadoop/spark/wal";
>
> jsc.checkpoint(checkpointDirectory);
>
>
> //I am using the My Receiver Based Consumer (
> https://github.com/dibbhatt/kafka-spark-consumer) . But
> KafkaUtil.CreateStream will also work
>
> JavaDStream unionStreams = ReceiverLauncher.launch(
> jsc, props, numberOfReceivers, StorageLevel.OFF_HEAP());
>
>
>
>
> Regards,
> Dibyendu
>
> On Sat, Sep 26, 2015 at 11:59 AM, N B <nb.nos...@gmail.com> wrote:
>
>> Hi Dibyendu,
>>
>> How does one go about configuring spark streaming to use tachyon as its
>> place for storing checkpoints? Also, can one do this with tachyon running
>> on a completely different node than where spark processes are running?
>>
>> Thanks
>> Nikunj
>>
>>
>> On Thu, May 21, 2015 at 8:35 PM, Dibyendu Bhattacharya <
>> dibyendu.bhattach...@gmail.com> wrote:
>>
>>> Hi Tathagata,
>>>
>>> Thanks for looking into this. Further investigating I found that the
>>> issue is with Tachyon does not support File Append. The streaming receiver
>>> which writes to WAL when failed, and again restarted, not able to append to
>>> same WAL file after restart.
>>>
>>> I raised this with Tachyon user group, and Haoyuan told that within 3
>>> months time Tachyon file append will be ready. Will revisit this issue
>>> again then .
>>>
>>> Regards,
>>> Dibyendu
>>>
>>>
>>> On Fri, May 22, 2015 at 12:24 AM, Tathagata Das <t...@databricks.com>
>>> wrote:
>>>
>>>> Looks like somehow the file size reported by the FSInputDStream of
>>>> Tachyon's FileSystem interface, is returning zero.
>>>>
>>>> On Mon, May 11, 2015 at 4:38 AM, Dibyendu Bhattacharya <
>>>> dibyendu.bhattach...@gmail.com> wrote:
>>>>
>>>>> Just to follow up this thread further .
>>>>>
>>>>> I was doing some fault tolerant testing of Spark Streaming with
>>>>> Tachyon as OFF_HEAP block store. As I said in earlier email, I could able
>>>>> to solve the BlockNotFound exception when I used Hierarchical Storage
>>>>> of Tachyon ,  which is good.
>>>>>
>>>>> I continue doing some testing around 

Re: Spark Streaming with Tachyon : Data Loss on Receiver Failure due to WAL error

[Dibyendu Bhattacharya]

Hi,

Recently I was working on a PR to use Tachyon as OFF_HEAP store for Spark
Streaming and make sure Spark Streaming can recover from Driver failure and
recover the blocks form Tachyon.

The The Motivation for this PR is  :

If Streaming application stores the blocks OFF_HEAP, it may not need any
WAL like feature to recover from Driver failure. As long as the writing of
blocks to Tachyon from Streaming receiver is durable, it should be
recoverable from Tachyon directly on Driver failure.
This can solve the issue of expensive WAL write and duplicating the blocks
both in MEMORY and also WAL and also guarantee end to end No-Data-Loss
channel using OFF_HEAP store.

https://github.com/apache/spark/pull/8817

This PR still under review . But having done various fail over testing in
my environment , I see this PR worked perfectly fine without any data loss
. Let see what TD and other have to say on this PR .

Below is the configuration I used to test this PR ..


Spark : 1.6 from Master
Tachyon : 0.7.1

SparkConfiguration Details :

SparkConf conf = new SparkConf().setAppName("TestTachyon")
.set("spark.streaming.unpersist", "true")
.set("spark.local.dir", "/mnt1/spark/tincan")
.set("tachyon.zookeeper.address","10.252.5.113:2182")
.set("tachyon.usezookeeper","true")
.set("spark.externalBlockStore.url", "tachyon-ft://
ip-10-252-5-113.asskickery.us:19998")
.set("spark.externalBlockStore.baseDir", "/sparkstreaming")
.set("spark.externalBlockStore.folderName","pearson")
.set("spark.externalBlockStore.dirId", "subpub")

.set("spark.externalBlockStore.keepExternalBlockStoreDirOnShutdown","true");

JavaStreamingContext jsc = new JavaStreamingContext(conf, new Duration(
1));

String checkpointDirectory = "hdfs://10.252.5.113:9000/user/hadoop/spark/wal
";

jsc.checkpoint(checkpointDirectory);


//I am using the My Receiver Based Consumer (
https://github.com/dibbhatt/kafka-spark-consumer) . But
KafkaUtil.CreateStream will also work

JavaDStream unionStreams = ReceiverLauncher.launch(
jsc, props, numberOfReceivers, StorageLevel.OFF_HEAP());




Regards,
Dibyendu

On Sat, Sep 26, 2015 at 11:59 AM, N B <nb.nos...@gmail.com> wrote:

> Hi Dibyendu,
>
> How does one go about configuring spark streaming to use tachyon as its
> place for storing checkpoints? Also, can one do this with tachyon running
> on a completely different node than where spark processes are running?
>
> Thanks
> Nikunj
>
>
> On Thu, May 21, 2015 at 8:35 PM, Dibyendu Bhattacharya <
> dibyendu.bhattach...@gmail.com> wrote:
>
>> Hi Tathagata,
>>
>> Thanks for looking into this. Further investigating I found that the
>> issue is with Tachyon does not support File Append. The streaming receiver
>> which writes to WAL when failed, and again restarted, not able to append to
>> same WAL file after restart.
>>
>> I raised this with Tachyon user group, and Haoyuan told that within 3
>> months time Tachyon file append will be ready. Will revisit this issue
>> again then .
>>
>> Regards,
>> Dibyendu
>>
>>
>> On Fri, May 22, 2015 at 12:24 AM, Tathagata Das <t...@databricks.com>
>> wrote:
>>
>>> Looks like somehow the file size reported by the FSInputDStream of
>>> Tachyon's FileSystem interface, is returning zero.
>>>
>>> On Mon, May 11, 2015 at 4:38 AM, Dibyendu Bhattacharya <
>>> dibyendu.bhattach...@gmail.com> wrote:
>>>
>>>> Just to follow up this thread further .
>>>>
>>>> I was doing some fault tolerant testing of Spark Streaming with Tachyon
>>>> as OFF_HEAP block store. As I said in earlier email, I could able to solve
>>>> the BlockNotFound exception when I used Hierarchical Storage of
>>>> Tachyon ,  which is good.
>>>>
>>>> I continue doing some testing around storing the Spark Streaming WAL
>>>> and CheckPoint files also in Tachyon . Here is few finding ..
>>>>
>>>>
>>>> When I store the Spark Streaming Checkpoint location in Tachyon , the
>>>> throughput is much higher . I tested the Driver and Receiver failure cases
>>>> , and Spark Streaming is able to recover without any Data Loss on Driver
>>>> failure.
>>>>
>>>> *But on Receiver failure , Spark Streaming looses data* as I see
>>>> Exception while reading the WAL file from Tachyon "receivedData" location
>>>>  for the same Receiver id which just failed.
>>>>
>>>> If I change the Checkpoint locati

Re: Spark Streaming with Tachyon : Data Loss on Receiver Failure due to WAL error

[N B]

Hi Dibyendu,

Thanks. I believe I understand why it has been an issue using S3 for
checkpoints based on your explanation. But does this limitation apply only
if recovery is needed in case of driver failure?

What if we are not interested in recovery after a driver failure. However,
just for the purposes of running streaming pipelines that do
reduceByKeyAndWindow() and updateStateByKey() operations it needs to have a
checkpoint directory configured.

Do you think this usage will also run into issues if an S3 location is
provided for the checkpoint directory. We will not use it to do any
explicit recovery like I stated above.

Thanks
Nikunj



On Sat, Sep 26, 2015 at 3:13 AM, Dibyendu Bhattacharya <
dibyendu.bhattach...@gmail.com> wrote:

> In Spark Streaming , Checkpoint Directory is used for two purpose
>
> 1. Metadata checkpointing
>
> 2. Data checkpointing
>
> If you enable WAL to recover from Driver failure, Spark Streaming will
> also write the Received Blocks in WAL which stored in checkpoint directory.
>
> For streaming solution to recover from any failure without any data loss ,
> you need to enable Meta Data Check pointing and WAL.  You do not need to
> enable Data Check pointing.
>
> From my experiments and the PR I mentioned , I configured the Meta Data
> Check Pointing in HDFS , and stored the Received Blocks OFF_HEAP.  And I
> did not use any WAL . The PR I proposed would recover from Driver fail-over
> without using any WAL like feature because Blocks are already available in
> Tachyon. The Meta Data Checkpoint helps to recover the meta data about past
> received blocks.
>
> Now the question is , can I configure Tachyon as my Metadata Checkpoint
> location ? I tried that , and Streaming application writes the
> receivedBlockMeataData to Tachyon, but on driver failure, it can not
> recover the received block meta data from Tachyon. I sometime see Zero size
> files in Tachyon checkpoint location , and it can not recover past events .
> I need to understand what is the issue of storing meta data in Tachyon .
> That needs a different JIRA I guess.
>
> Let me know I am able to explain the current scenario around Spark
> Streaming and Tachyon .
>
> Regards,
> Dibyendu
>
>
>
>
> On Sat, Sep 26, 2015 at 1:04 PM, N B <nb.nos...@gmail.com> wrote:
>
>> Hi Dibyendu,
>>
>> I am not sure I understand completely. But are you suggesting that
>> currently there is no way to enable Checkpoint directory to be in Tachyon?
>>
>> Thanks
>> Nikunj
>>
>>
>> On Fri, Sep 25, 2015 at 11:49 PM, Dibyendu Bhattacharya <
>> dibyendu.bhattach...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Recently I was working on a PR to use Tachyon as OFF_HEAP store for
>>> Spark Streaming and make sure Spark Streaming can recover from Driver
>>> failure and recover the blocks form Tachyon.
>>>
>>> The The Motivation for this PR is  :
>>>
>>> If Streaming application stores the blocks OFF_HEAP, it may not need any
>>> WAL like feature to recover from Driver failure. As long as the writing of
>>> blocks to Tachyon from Streaming receiver is durable, it should be
>>> recoverable from Tachyon directly on Driver failure.
>>> This can solve the issue of expensive WAL write and duplicating the
>>> blocks both in MEMORY and also WAL and also guarantee end to end
>>> No-Data-Loss channel using OFF_HEAP store.
>>>
>>> https://github.com/apache/spark/pull/8817
>>>
>>> This PR still under review . But having done various fail over testing
>>> in my environment , I see this PR worked perfectly fine without any data
>>> loss . Let see what TD and other have to say on this PR .
>>>
>>> Below is the configuration I used to test this PR ..
>>>
>>>
>>> Spark : 1.6 from Master
>>> Tachyon : 0.7.1
>>>
>>> SparkConfiguration Details :
>>>
>>> SparkConf conf = new SparkConf().setAppName("TestTachyon")
>>> .set("spark.streaming.unpersist", "true")
>>> .set("spark.local.dir", "/mnt1/spark/tincan")
>>> .set("tachyon.zookeeper.address","10.252.5.113:2182")
>>> .set("tachyon.usezookeeper","true")
>>> .set("spark.externalBlockStore.url", "tachyon-ft://
>>> ip-10-252-5-113.asskickery.us:19998")
>>> .set("spark.externalBlockStore.baseDir", "/sparkstreaming")
>>> .set("spark.externalBlockStore.folderName","pearson")
>>> .set("spark.extern

Re: Spark Streaming with Tachyon : Data Loss on Receiver Failure due to WAL error

[N B]

I wanted to add that we are not configuring the WAL in our scenario.

Thanks again,
Nikunj


On Sat, Sep 26, 2015 at 11:35 AM, N B <nb.nos...@gmail.com> wrote:

> Hi Dibyendu,
>
> Thanks. I believe I understand why it has been an issue using S3 for
> checkpoints based on your explanation. But does this limitation apply only
> if recovery is needed in case of driver failure?
>
> What if we are not interested in recovery after a driver failure. However,
> just for the purposes of running streaming pipelines that do
> reduceByKeyAndWindow() and updateStateByKey() operations it needs to have a
> checkpoint directory configured.
>
> Do you think this usage will also run into issues if an S3 location is
> provided for the checkpoint directory. We will not use it to do any
> explicit recovery like I stated above.
>
> Thanks
> Nikunj
>
>
>
> On Sat, Sep 26, 2015 at 3:13 AM, Dibyendu Bhattacharya <
> dibyendu.bhattach...@gmail.com> wrote:
>
>> In Spark Streaming , Checkpoint Directory is used for two purpose
>>
>> 1. Metadata checkpointing
>>
>> 2. Data checkpointing
>>
>> If you enable WAL to recover from Driver failure, Spark Streaming will
>> also write the Received Blocks in WAL which stored in checkpoint directory.
>>
>> For streaming solution to recover from any failure without any data loss
>> , you need to enable Meta Data Check pointing and WAL.  You do not need to
>> enable Data Check pointing.
>>
>> From my experiments and the PR I mentioned , I configured the Meta Data
>> Check Pointing in HDFS , and stored the Received Blocks OFF_HEAP.  And I
>> did not use any WAL . The PR I proposed would recover from Driver fail-over
>> without using any WAL like feature because Blocks are already available in
>> Tachyon. The Meta Data Checkpoint helps to recover the meta data about past
>> received blocks.
>>
>> Now the question is , can I configure Tachyon as my Metadata Checkpoint
>> location ? I tried that , and Streaming application writes the
>> receivedBlockMeataData to Tachyon, but on driver failure, it can not
>> recover the received block meta data from Tachyon. I sometime see Zero size
>> files in Tachyon checkpoint location , and it can not recover past events .
>> I need to understand what is the issue of storing meta data in Tachyon .
>> That needs a different JIRA I guess.
>>
>> Let me know I am able to explain the current scenario around Spark
>> Streaming and Tachyon .
>>
>> Regards,
>> Dibyendu
>>
>>
>>
>>
>> On Sat, Sep 26, 2015 at 1:04 PM, N B <nb.nos...@gmail.com> wrote:
>>
>>> Hi Dibyendu,
>>>
>>> I am not sure I understand completely. But are you suggesting that
>>> currently there is no way to enable Checkpoint directory to be in Tachyon?
>>>
>>> Thanks
>>> Nikunj
>>>
>>>
>>> On Fri, Sep 25, 2015 at 11:49 PM, Dibyendu Bhattacharya <
>>> dibyendu.bhattach...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Recently I was working on a PR to use Tachyon as OFF_HEAP store for
>>>> Spark Streaming and make sure Spark Streaming can recover from Driver
>>>> failure and recover the blocks form Tachyon.
>>>>
>>>> The The Motivation for this PR is  :
>>>>
>>>> If Streaming application stores the blocks OFF_HEAP, it may not need
>>>> any WAL like feature to recover from Driver failure. As long as the writing
>>>> of blocks to Tachyon from Streaming receiver is durable, it should be
>>>> recoverable from Tachyon directly on Driver failure.
>>>> This can solve the issue of expensive WAL write and duplicating the
>>>> blocks both in MEMORY and also WAL and also guarantee end to end
>>>> No-Data-Loss channel using OFF_HEAP store.
>>>>
>>>> https://github.com/apache/spark/pull/8817
>>>>
>>>> This PR still under review . But having done various fail over testing
>>>> in my environment , I see this PR worked perfectly fine without any data
>>>> loss . Let see what TD and other have to say on this PR .
>>>>
>>>> Below is the configuration I used to test this PR ..
>>>>
>>>>
>>>> Spark : 1.6 from Master
>>>> Tachyon : 0.7.1
>>>>
>>>> SparkConfiguration Details :
>>>>
>>>> SparkConf conf = new SparkConf().setAppName("TestTachyon")
>>>> .set("spark.streaming.unpersist", "true")

BlockNotFoundException when running spark word count on Tachyon

[Todd]

I am using tachyon in the spark program below,but I encounter a 
BlockNotFoundxception.
Does someone know what's wrong and also is there guide on how to configure 
spark to work with Tackyon?Thanks!

conf.set(spark.externalBlockStore.url, tachyon://10.18.19.33:19998)
conf.set(spark.externalBlockStore.baseDir,/spark)
val sc = new SparkContext(conf)
import org.apache.spark.storage.StorageLevel
val rdd = sc.parallelize(List(1, 2, 3, 4, 5, 6))
rdd.persist(StorageLevel.OFF_HEAP)
val count = rdd.count()
   val sum = rdd.reduce(_ + _)
println(sThe count: $count, The sum is: $sum)


15/08/26 14:52:03 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have 
all completed, from pool
org.apache.spark.SparkException: Job aborted due to stage failure: Task 5 in 
stage 0.0 failed 1 times, most recent failure: Lost task 5.0 in stage 0.0 (TID 
5, localhost): java.lang.RuntimeException: 
org.apache.spark.storage.BlockNotFoundException: Block rdd_0_5 not found
at 
org.apache.spark.storage.BlockManager.getBlockData(BlockManager.scala:308)
at 
org.apache.spark.network.netty.NettyBlockRpcServer$$anonfun$2.apply(NettyBlockRpcServer.scala:57)
at 
org.apache.spark.network.netty.NettyBlockRpcServer$$anonfun$2.apply(NettyBlockRpcServer.scala:57)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at 
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
at 
org.apache.spark.network.netty.NettyBlockRpcServer.receive(NettyBlockRpcServer.scala:57)
at 
org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:114)
at 
org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:87)
at 
org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:101)
at 
org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51)
at 
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at 
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at 
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at 
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at 
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
at 
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)

Re: BlockNotFoundException when running spark word count on Tachyon

[Dibyendu Bhattacharya]

Sometime back I was playing with Spark and Tachyon and I also found this
issue .  The issue here is TachyonBlockManager put the blocks in
WriteType.TRY_CACHE configuration . And because of this Blocks ate evicted
from Tachyon Cache when Memory is full and when Spark try to find the block
it throws  BlockNotFoundException .

To solve this I tried Hierarchical Storage on Tachyon ( http://tachyon
-project.org/Hierarchy-Storage-on-Tachyon.html ) , and that seems to have
worked and I did not see any any Spark Job failed due to
BlockNotFoundException.
below is my  Hierarchical Storage settings which I used..

  -Dtachyon.worker.hierarchystore.level.max=2
  -Dtachyon.worker.hierarchystore.level0.alias=MEM
  -Dtachyon.worker.hierarchystore.level0.dirs.path=$TACHYON_RAM_FOLDER

-Dtachyon.worker.hierarchystore.level0.dirs.quota=$TACHYON_WORKER_MEMORY_SIZE
  -Dtachyon.worker.hierarchystore.level1.alias=HDD
  -Dtachyon.worker.hierarchystore.level1.dirs.path=/mnt/tachyon
  -Dtachyon.worker.hierarchystore.level1.dirs.quota=50GB
  -Dtachyon.worker.allocate.strategy=MAX_FREE
  -Dtachyon.worker.evict.strategy=LRU

Regards,
Dibyendu

On Wed, Aug 26, 2015 at 12:25 PM, Todd bit1...@163.com wrote:


 I am using tachyon in the spark program below,but I encounter a
 BlockNotFoundxception.
 Does someone know what's wrong and also is there guide on how to configure
 spark to work with Tackyon?Thanks!

 conf.set(spark.externalBlockStore.url, tachyon://10.18.19.33:19998
 )
 conf.set(spark.externalBlockStore.baseDir,/spark)
 val sc = new SparkContext(conf)
 import org.apache.spark.storage.StorageLevel
 val rdd = sc.parallelize(List(1, 2, 3, 4, 5, 6))
 rdd.persist(StorageLevel.OFF_HEAP)
 val count = rdd.count()
val sum = rdd.reduce(_ + _)
 println(sThe count: $count, The sum is: $sum)


 15/08/26 14:52:03 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks
 have all completed, from pool
 org.apache.spark.SparkException: Job aborted due to stage failure: Task 5
 in stage 0.0 failed 1 times, most recent failure: Lost task 5.0 in stage
 0.0 (TID 5, localhost): java.lang.RuntimeException:
 org.apache.spark.storage.BlockNotFoundException: Block rdd_0_5 not found
 at
 org.apache.spark.storage.BlockManager.getBlockData(BlockManager.scala:308)
 at
 org.apache.spark.network.netty.NettyBlockRpcServer$$anonfun$2.apply(NettyBlockRpcServer.scala:57)
 at
 org.apache.spark.network.netty.NettyBlockRpcServer$$anonfun$2.apply(NettyBlockRpcServer.scala:57)
 at
 scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
 at
 scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
 at
 scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
 at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
 at
 scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
 at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
 at
 org.apache.spark.network.netty.NettyBlockRpcServer.receive(NettyBlockRpcServer.scala:57)
 at
 org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:114)
 at
 org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:87)
 at
 org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:101)
 at
 org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51)
 at
 io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
 at
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
 at
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
 at
 io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
 at
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
 at
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
 at
 io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
 at
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
 at
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
 at
 io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
 at
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
 at
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
 at
 io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
 at
 io.netty.channel.nio.AbstractNioByteChannel

Re: BlockNotFoundException when running spark word count on Tachyon

[Dibyendu Bhattacharya]

The URL seems to have changed .. here is the one ..
http://tachyon-project.org/documentation/Tiered-Storage-on-Tachyon.html



On Wed, Aug 26, 2015 at 12:32 PM, Dibyendu Bhattacharya 
dibyendu.bhattach...@gmail.com wrote:

 Sometime back I was playing with Spark and Tachyon and I also found this
 issue .  The issue here is TachyonBlockManager put the blocks in
 WriteType.TRY_CACHE configuration . And because of this Blocks ate evicted
 from Tachyon Cache when Memory is full and when Spark try to find the
 block it throws  BlockNotFoundException .

 To solve this I tried Hierarchical Storage on Tachyon ( http://tachyon
 -project.org/Hierarchy-Storage-on-Tachyon.html ) , and that seems to have
 worked and I did not see any any Spark Job failed due to 
 BlockNotFoundException.
 below is my  Hierarchical Storage settings which I used..

   -Dtachyon.worker.hierarchystore.level.max=2
   -Dtachyon.worker.hierarchystore.level0.alias=MEM
   -Dtachyon.worker.hierarchystore.level0.dirs.path=$TACHYON_RAM_FOLDER

 -Dtachyon.worker.hierarchystore.level0.dirs.quota=$TACHYON_WORKER_MEMORY_SIZE
   -Dtachyon.worker.hierarchystore.level1.alias=HDD
   -Dtachyon.worker.hierarchystore.level1.dirs.path=/mnt/tachyon
   -Dtachyon.worker.hierarchystore.level1.dirs.quota=50GB
   -Dtachyon.worker.allocate.strategy=MAX_FREE
   -Dtachyon.worker.evict.strategy=LRU

 Regards,
 Dibyendu

 On Wed, Aug 26, 2015 at 12:25 PM, Todd bit1...@163.com wrote:


 I am using tachyon in the spark program below,but I encounter a
 BlockNotFoundxception.
 Does someone know what's wrong and also is there guide on how to
 configure spark to work with Tackyon?Thanks!

 conf.set(spark.externalBlockStore.url, tachyon://10.18.19.33:19998
 )
 conf.set(spark.externalBlockStore.baseDir,/spark)
 val sc = new SparkContext(conf)
 import org.apache.spark.storage.StorageLevel
 val rdd = sc.parallelize(List(1, 2, 3, 4, 5, 6))
 rdd.persist(StorageLevel.OFF_HEAP)
 val count = rdd.count()
val sum = rdd.reduce(_ + _)
 println(sThe count: $count, The sum is: $sum)


 15/08/26 14:52:03 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose
 tasks have all completed, from pool
 org.apache.spark.SparkException: Job aborted due to stage failure: Task 5
 in stage 0.0 failed 1 times, most recent failure: Lost task 5.0 in stage
 0.0 (TID 5, localhost): java.lang.RuntimeException:
 org.apache.spark.storage.BlockNotFoundException: Block rdd_0_5 not found
 at
 org.apache.spark.storage.BlockManager.getBlockData(BlockManager.scala:308)
 at
 org.apache.spark.network.netty.NettyBlockRpcServer$$anonfun$2.apply(NettyBlockRpcServer.scala:57)
 at
 org.apache.spark.network.netty.NettyBlockRpcServer$$anonfun$2.apply(NettyBlockRpcServer.scala:57)
 at
 scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
 at
 scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
 at
 scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
 at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
 at
 scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
 at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
 at
 org.apache.spark.network.netty.NettyBlockRpcServer.receive(NettyBlockRpcServer.scala:57)
 at
 org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:114)
 at
 org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:87)
 at
 org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:101)
 at
 org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51)
 at
 io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
 at
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
 at
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
 at
 io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
 at
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
 at
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
 at
 io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
 at
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
 at
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
 at
 io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
 at
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333

Re: Spark is in-memory processing, how then can Tachyon make Spark faster?

[andy petrella]

Exactly!

The sharing part is used in the Spark Notebook (this one
https://github.com/andypetrella/spark-notebook/blob/master/notebooks/Tachyon%20Test.snb)
so we can share stuffs between notebooks which are different SparkContext
(in diff JVM).

OTOH, we have a project that creates micro services on genomics data, for
several reasons we used Tachyon to server genomes cubes (ranges across
genomes), see here https://github.com/med-at-scale/high-health.

HTH
andy

On Fri, Aug 7, 2015 at 8:36 PM Calvin Jia jia.cal...@gmail.com wrote:

 Hi,

 Tachyon http://tachyon-project.org manages memory off heap which can
 help prevent long GC pauses. Also, using Tachyon will allow the data to be
 shared between Spark jobs if they use the same dataset.

 Here's http://www.meetup.com/Tachyon/events/222485713/ a production use
 case where Baidu runs Tachyon to get 30x performance improvement in their
 SparkSQL workload.

 Hope this helps,
 Calvin

 On Fri, Aug 7, 2015 at 9:42 AM, Muler mulugeta.abe...@gmail.com wrote:

 Spark is an in-memory engine and attempts to do computation in-memory.
 Tachyon is memory-centeric distributed storage, OK, but how would that help
 ran Spark faster?


 --
andy

Re: tachyon

[Abhishek R. Singh]

Thanks Calvin - much appreciated !

-Abhishek-

On Aug 7, 2015, at 11:11 AM, Calvin Jia jia.cal...@gmail.com wrote:

 Hi Abhishek,
 
 Here's a production use case that may interest you: 
 http://www.meetup.com/Tachyon/events/222485713/
 
 Baidu is using Tachyon to manage more than 100 nodes in production resulting 
 in a 30x performance improvement for their SparkSQL workload. They are also 
 using the tiered storage feature in Tachyon giving them over 2PB of Tachyon 
 managed space.
 
 Hope this helps,
 Calvin
 
 On Fri, Aug 7, 2015 at 10:00 AM, Ted Yu yuzhih...@gmail.com wrote:
 Looks like you would get better response on Tachyon's mailing list:
 
 https://groups.google.com/forum/?fromgroups#!forum/tachyon-users
 
 Cheers
 
 On Fri, Aug 7, 2015 at 9:56 AM, Abhishek R. Singh 
 abhis...@tetrationanalytics.com wrote:
 Do people use Tachyon in production, or is it experimental grade still?
 
 Regards,
 Abhishek
 
 -
 To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
 For additional commands, e-mail: user-h...@spark.apache.org
 
 
 

Spark is in-memory processing, how then can Tachyon make Spark faster?

[Muler]

Spark is an in-memory engine and attempts to do computation in-memory.
Tachyon is memory-centeric distributed storage, OK, but how would that help
ran Spark faster?

Re: tachyon

[Ted Yu]

Looks like you would get better response on Tachyon's mailing list:

https://groups.google.com/forum/?fromgroups#!forum/tachyon-users

Cheers

On Fri, Aug 7, 2015 at 9:56 AM, Abhishek R. Singh 
abhis...@tetrationanalytics.com wrote:

 Do people use Tachyon in production, or is it experimental grade still?

 Regards,
 Abhishek

 -
 To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
 For additional commands, e-mail: user-h...@spark.apache.org

tachyon

[Abhishek R. Singh]

Do people use Tachyon in production, or is it experimental grade still?

Regards,
Abhishek

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Spark is in-memory processing, how then can Tachyon make Spark faster?

[Calvin Jia]

Hi,

Tachyon http://tachyon-project.org manages memory off heap which can help
prevent long GC pauses. Also, using Tachyon will allow the data to be
shared between Spark jobs if they use the same dataset.

Here's http://www.meetup.com/Tachyon/events/222485713/ a production use
case where Baidu runs Tachyon to get 30x performance improvement in their
SparkSQL workload.

Hope this helps,
Calvin

On Fri, Aug 7, 2015 at 9:42 AM, Muler mulugeta.abe...@gmail.com wrote:

 Spark is an in-memory engine and attempts to do computation in-memory.
 Tachyon is memory-centeric distributed storage, OK, but how would that help
 ran Spark faster?

Re: How can I use Tachyon with SPARK?

[Himanshu Mehra]

Hi June,

As i understand your problem, you are running spark 1.3 and want to use
Tachyon with it. what you need to do is simply build the latest Spark and
Tachyon and set some configuration is Spark. In fact spark 1.3 has
spark/core/pom.xm, you have to find the core folder in your spark home
and inside it there must be a pom.xml file, if its not there i suspect
your spark code is broken or incomplete, and you should download the source
code again. what you need to do is find the Tachyon dependency in that
pom.xml and set its version as per you want. plus there are some additional
configuration you need to do in spark to enable Tachyon with spark. here is
list of sparkConf properties regarding Tachyon :
https://docs.sigmoidanalytics.com/index.php/Configuration_Settings

And if you find any difficulty building or recompiling spark you should
refer to this doc : https://spark.apache.org/docs/1.3.0/building-spark.html

for any further assistance please write back.

Thank you
Himashu



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/How-can-I-use-Tachyon-with-SPARK-tp23312p23317.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Spark Streaming with Tachyon : Data Loss on Receiver Failure due to WAL error

[Tathagata Das]

Looks like somehow the file size reported by the FSInputDStream of
Tachyon's FileSystem interface, is returning zero.

On Mon, May 11, 2015 at 4:38 AM, Dibyendu Bhattacharya 
dibyendu.bhattach...@gmail.com wrote:

 Just to follow up this thread further .

 I was doing some fault tolerant testing of Spark Streaming with Tachyon as
 OFF_HEAP block store. As I said in earlier email, I could able to solve the
 BlockNotFound exception when I used Hierarchical Storage of Tachyon ,
  which is good.

 I continue doing some testing around storing the Spark Streaming WAL and
 CheckPoint files also in Tachyon . Here is few finding ..


 When I store the Spark Streaming Checkpoint location in Tachyon , the
 throughput is much higher . I tested the Driver and Receiver failure cases
 , and Spark Streaming is able to recover without any Data Loss on Driver
 failure.

 *But on Receiver failure , Spark Streaming looses data* as I see
 Exception while reading the WAL file from Tachyon receivedData location
  for the same Receiver id which just failed.

 If I change the Checkpoint location back to HDFS , Spark Streaming can
 recover from both Driver and Receiver failure .

 Here is the Log details when Spark Streaming receiver failed ...I raised a
 JIRA for the same issue : https://issues.apache.org/jira/browse/SPARK-7525



 INFO : org.apache.spark.scheduler.DAGScheduler - *Executor lost: 2 (epoch
 1)*
 INFO : org.apache.spark.storage.BlockManagerMasterEndpoint - Trying to
 remove executor 2 from BlockManagerMaster.
 INFO : org.apache.spark.storage.BlockManagerMasterEndpoint - Removing
 block manager BlockManagerId(2, 10.252.5.54, 45789)
 INFO : org.apache.spark.storage.BlockManagerMaster - Removed 2
 successfully in removeExecutor
 INFO : org.apache.spark.streaming.scheduler.ReceiverTracker - *Registered
 receiver for stream 2 from 10.252.5.62*:47255
 WARN : org.apache.spark.scheduler.TaskSetManager - Lost task 2.1 in stage
 103.0 (TID 421, 10.252.5.62): org.apache.spark.SparkException: *Could not
 read data from write ahead log record
 FileBasedWriteAheadLogSegment(tachyon-ft://10.252.5.113:19998/tachyon/checkpoint/receivedData/2/log-1431341091711-1431341151711,645603894,10891919
 http://10.252.5.113:19998/tachyon/checkpoint/receivedData/2/log-1431341091711-1431341151711,645603894,10891919)*
 at org.apache.spark.streaming.rdd.WriteAheadLogBackedBlockRDD.org
 $apache$spark$streaming$rdd$WriteAheadLogBackedBlockRDD$$getBlockFromWriteAheadLog$1(WriteAheadLogBackedBlockRDD.scala:144)
 at
 org.apache.spark.streaming.rdd.WriteAheadLogBackedBlockRDD$$anonfun$compute$1.apply(WriteAheadLogBackedBlockRDD.scala:168)
 at
 org.apache.spark.streaming.rdd.WriteAheadLogBackedBlockRDD$$anonfun$compute$1.apply(WriteAheadLogBackedBlockRDD.scala:168)
 at scala.Option.getOrElse(Option.scala:120)
 at
 org.apache.spark.streaming.rdd.WriteAheadLogBackedBlockRDD.compute(WriteAheadLogBackedBlockRDD.scala:168)
 at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
 at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
 at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:87)
 at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
 at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
 at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63)
 at org.apache.spark.scheduler.Task.run(Task.scala:70)
 at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
 at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:744)
 Caused by: java.lang.IllegalArgumentException:* Seek position is past
 EOF: 645603894, fileSize = 0*
 at tachyon.hadoop.HdfsFileInputStream.seek(HdfsFileInputStream.java:239)
 at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:37)
 at
 org.apache.spark.streaming.util.FileBasedWriteAheadLogRandomReader.read(FileBasedWriteAheadLogRandomReader.scala:37)
 at
 org.apache.spark.streaming.util.FileBasedWriteAheadLog.read(FileBasedWriteAheadLog.scala:104)
 at org.apache.spark.streaming.rdd.WriteAheadLogBackedBlockRDD.org
 $apache$spark$streaming$rdd$WriteAheadLogBackedBlockRDD$$getBlockFromWriteAheadLog$1(WriteAheadLogBackedBlockRDD.scala:141)
 ... 15 more

 INFO : org.apache.spark.scheduler.TaskSetManager - Starting task 2.2 in
 stage 103.0 (TID 422, 10.252.5.61, ANY, 1909 bytes)
 INFO : org.apache.spark.scheduler.TaskSetManager - Lost task 2.2 in stage
 103.0 (TID 422) on executor 10.252.5.61: org.apache.spark.SparkException
 (Could not read data from write ahead log record
 FileBasedWriteAheadLogSegment(tachyon-ft://
 10.252.5.113:19998/tachyon/checkpoint/receivedData/2/log-1431341091711-1431341151711,645603894,10891919))
 [duplicate 1]
 INFO : org.apache.spark.scheduler.TaskSetManager - Starting task 2.3 in
 stage 103.0 (TID 423, 10.252.5.62, ANY, 1909 bytes)
 INFO : org.apache.spark.deploy.client.AppClient$ClientActor

Re: Spark Streaming with Tachyon : Data Loss on Receiver Failure due to WAL error

[Dibyendu Bhattacharya]

Hi Tathagata,

Thanks for looking into this. Further investigating I found that the issue
is with Tachyon does not support File Append. The streaming receiver which
writes to WAL when failed, and again restarted, not able to append to same
WAL file after restart.

I raised this with Tachyon user group, and Haoyuan told that within 3
months time Tachyon file append will be ready. Will revisit this issue
again then .

Regards,
Dibyendu


On Fri, May 22, 2015 at 12:24 AM, Tathagata Das t...@databricks.com wrote:

 Looks like somehow the file size reported by the FSInputDStream of
 Tachyon's FileSystem interface, is returning zero.

 On Mon, May 11, 2015 at 4:38 AM, Dibyendu Bhattacharya 
 dibyendu.bhattach...@gmail.com wrote:

 Just to follow up this thread further .

 I was doing some fault tolerant testing of Spark Streaming with Tachyon
 as OFF_HEAP block store. As I said in earlier email, I could able to solve
 the BlockNotFound exception when I used Hierarchical Storage of Tachyon
 ,  which is good.

 I continue doing some testing around storing the Spark Streaming WAL and
 CheckPoint files also in Tachyon . Here is few finding ..


 When I store the Spark Streaming Checkpoint location in Tachyon , the
 throughput is much higher . I tested the Driver and Receiver failure cases
 , and Spark Streaming is able to recover without any Data Loss on Driver
 failure.

 *But on Receiver failure , Spark Streaming looses data* as I see
 Exception while reading the WAL file from Tachyon receivedData location
  for the same Receiver id which just failed.

 If I change the Checkpoint location back to HDFS , Spark Streaming can
 recover from both Driver and Receiver failure .

 Here is the Log details when Spark Streaming receiver failed ...I raised
 a JIRA for the same issue :
 https://issues.apache.org/jira/browse/SPARK-7525



 INFO : org.apache.spark.scheduler.DAGScheduler - *Executor lost: 2
 (epoch 1)*
 INFO : org.apache.spark.storage.BlockManagerMasterEndpoint - Trying to
 remove executor 2 from BlockManagerMaster.
 INFO : org.apache.spark.storage.BlockManagerMasterEndpoint - Removing
 block manager BlockManagerId(2, 10.252.5.54, 45789)
 INFO : org.apache.spark.storage.BlockManagerMaster - Removed 2
 successfully in removeExecutor
 INFO : org.apache.spark.streaming.scheduler.ReceiverTracker - *Registered
 receiver for stream 2 from 10.252.5.62*:47255
 WARN : org.apache.spark.scheduler.TaskSetManager - Lost task 2.1 in stage
 103.0 (TID 421, 10.252.5.62): org.apache.spark.SparkException: *Could
 not read data from write ahead log record
 FileBasedWriteAheadLogSegment(tachyon-ft://10.252.5.113:19998/tachyon/checkpoint/receivedData/2/log-1431341091711-1431341151711,645603894,10891919
 http://10.252.5.113:19998/tachyon/checkpoint/receivedData/2/log-1431341091711-1431341151711,645603894,10891919)*
 at org.apache.spark.streaming.rdd.WriteAheadLogBackedBlockRDD.org
 $apache$spark$streaming$rdd$WriteAheadLogBackedBlockRDD$$getBlockFromWriteAheadLog$1(WriteAheadLogBackedBlockRDD.scala:144)
 at
 org.apache.spark.streaming.rdd.WriteAheadLogBackedBlockRDD$$anonfun$compute$1.apply(WriteAheadLogBackedBlockRDD.scala:168)
 at
 org.apache.spark.streaming.rdd.WriteAheadLogBackedBlockRDD$$anonfun$compute$1.apply(WriteAheadLogBackedBlockRDD.scala:168)
 at scala.Option.getOrElse(Option.scala:120)
 at
 org.apache.spark.streaming.rdd.WriteAheadLogBackedBlockRDD.compute(WriteAheadLogBackedBlockRDD.scala:168)
 at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
 at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
 at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:87)
 at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
 at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
 at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63)
 at org.apache.spark.scheduler.Task.run(Task.scala:70)
 at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
 at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:744)
 Caused by: java.lang.IllegalArgumentException:* Seek position is past
 EOF: 645603894, fileSize = 0*
 at tachyon.hadoop.HdfsFileInputStream.seek(HdfsFileInputStream.java:239)
 at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:37)
 at
 org.apache.spark.streaming.util.FileBasedWriteAheadLogRandomReader.read(FileBasedWriteAheadLogRandomReader.scala:37)
 at
 org.apache.spark.streaming.util.FileBasedWriteAheadLog.read(FileBasedWriteAheadLog.scala:104)
 at org.apache.spark.streaming.rdd.WriteAheadLogBackedBlockRDD.org
 $apache$spark$streaming$rdd$WriteAheadLogBackedBlockRDD$$getBlockFromWriteAheadLog$1(WriteAheadLogBackedBlockRDD.scala:141)
 ... 15 more

 INFO : org.apache.spark.scheduler.TaskSetManager - Starting task 2.2 in
 stage 103.0 (TID 422, 10.252.5.61, ANY, 1909 bytes)
 INFO

Fast big data analytics with Spark on Tachyon in Baidu

[Haoyuan Li]

Dear all,

We’re organizing a meetup http://www.meetup.com/Tachyon/events/222485713/ on
May 28th at IBM in Forster City that might be of interest to the Spark
community. The focus is a production use case of Spark and Tachyon at Baidu.

You can sign up here: http://www.meetup.com/Tachyon/events/222485713/

Hope some of you can make it!

Best,

Haoyuan

Re: Spark SQL 1.3.1 saveAsParquetFile will output tachyon file with different block size

[Calvin Jia]

Hi,

You can apply this patch https://github.com/apache/spark/pull/5354 and
recompile.

Hope this helps,
Calvin

On Tue, Apr 28, 2015 at 1:19 PM, sara mustafa eng.sara.must...@gmail.com
wrote:

 Hi Zhang,

 How did you compile Spark 1.3.1 with Tachyon? when i changed Tachyon
 version
 to 0.6.3 in core/pom.xml, make-distribution.sh and try to compile again,
 many compilation errors raised.

 Thanks,




 --
 View this message in context:
 http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-SQL-1-3-1-saveAsParquetFile-will-output-tachyon-file-with-different-block-size-tp11561p11870.html
 Sent from the Apache Spark Developers List mailing list archive at
 Nabble.com.

 -
 To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
 For additional commands, e-mail: dev-h...@spark.apache.org

tachyon on machines launched with spark-ec2 scripts

[Daniel Mahler]

I have a cluster launched with spark-ec2.
I can see a TachyonMaster process running,
but I do not seem to be able to use tachyon from the spark-shell.

if I try

rdd.saveAsTextFile(tachyon://localhost:19998/path)
I get

15/04/24 19:18:31 INFO TaskSetManager: Starting task 12.2 in stage 1.0 (TID
216, ip-10-63-69-48.ec2.internal, PROCESS_LOCAL, 1383 bytes)
15/04/24 19:18:31 WARN TaskSetManager: Lost task 32.2 in stage 1.0 (TID
177, ip-10-63-69-48.ec2.internal): java.io.IOException: Failed to connect
to master localhost/127.0.0.1:19998 after 5 attempts
at tachyon.client.TachyonFS.connect(TachyonFS.java:293)
at tachyon.client.TachyonFS.getUnderfsAddress(TachyonFS.java:1224)
at tachyon.hadoop.TFS.initialize(TFS.java:289)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2262)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:86)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2296)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2278)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:316)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:194)
at
org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:83)
at
org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1068)
at
org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1059)
at java.lang.Thread.run(Thread.java:745)
Caused by: tachyon.org.apache.thrift.TException: Failed to connect to
master localhost/127.0.0.1:19998 after 5 attempts
at tachyon.master.MasterClient.connect(MasterClient.java:178)
at tachyon.client.TachyonFS.connect(TachyonFS.java:290)
... 17 more
Caused by: tachyon.org.apache.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused
at
tachyon.org.apache.thrift.transport.TSocket.open(TSocket.java:185)
at
tachyon.org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
at tachyon.master.MasterClient.connect(MasterClient.java:156)
... 18 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at
tachyon.org.apache.thrift.transport.TSocket.open(TSocket.java:180)
... 20 more


 What do I need to do before I can use tachyon?

thanks
Daniel

Re: Why does the HDFS parquet file generated by Spark SQL have different size with those on Tachyon?

[Reynold Xin]

It's because you did a repartition -- which rearranges all the data.

Parquet uses all kinds of compression techniques such as dictionary
encoding and run-length encoding, which would result in the size difference
when the data is ordered different.

On Fri, Apr 17, 2015 at 4:51 AM, zhangxiongfei zhangxiongfei0...@163.com
wrote:

 Hi,
 I did some tests on Parquet Files with Spark SQL DataFrame API.
 I generated 36 gzip compressed parquet files by Spark SQL and stored them
 on Tachyon,The size of each file is about  222M.Then read them with below
 code.
 val tfs
 =sqlContext.parquetFile(tachyon://datanode8.bitauto.dmp:19998/apps/tachyon/adClick);
 Next,I just save this DataFrame onto HDFS with below code.It will generate
 36 parquet files too,but the size of each file is about 265M

 tfs.repartition(36).saveAsParquetFile(/user/zhangxf/adClick-parquet-tachyon);
 My question is Why the files on HDFS has different size with those on
 Tachyon even though they come from the same original data?


 Thanks
 Zhang Xiongfei

Re: Spark SQL 1.3.1 saveAsParquetFile will output tachyon file with different block size

[Cheng Lian]

Would you mind to open a JIRA for this?

I think your suspicion makes sense. Will have a look at this tomorrow. 
Thanks for reporting!


Cheng

On 4/13/15 7:13 PM, zhangxiongfei wrote:

Hi experts
I run below code  in Spark Shell to access parquet files in Tachyon.
1.First,created a DataFrame by loading a bunch of Parquet Files in Tachyon
val ta3 
=sqlContext.parquetFile(tachyon://tachyonserver:19998/apps/tachyon/zhangxf/parquetAdClick-6p-256m);
2.Second, set the fs.local.block.size to 256M to make sure that block size of 
output files in Tachyon is 256M.
sc.hadoopConfiguration.setLong(fs.local.block.size,268435456)
3.Third,saved above DataFrame into Parquet files that is stored in Tachyon
   
ta3.saveAsParquetFile(tachyon://tachyonserver:19998/apps/tachyon/zhangxf/parquetAdClick-6p-256m-test);
After above code run successfully, the output parquet files were stored in Tachyon,but 
these files have different block size,below is the information of those files in the path 
tachyon://tachyonserver:19998/apps/tachyon/zhangxf/parquetAdClick-6p-256m-test:
 File Name Size  Block Size In-Memory   
  Pin Creation Time
  _SUCCESS  0.00 B   256.00 MB 100% NO  
   04-13-2015 17:48:23:519
_common_metadata  1088.00 B  256.00 MB 100% NO 
04-13-2015 17:48:23:741
_metadata   22.71 KB   256.00 MB 100% NO
 04-13-2015 17:48:23:646
part-r-1.parquet 177.19 MB 32.00 MB  100% NO 
04-13-2015 17:46:44:626
part-r-2.parquet 177.21 MB 32.00 MB  100% NO 
04-13-2015 17:46:44:636
part-r-3.parquet 177.02 MB 32.00 MB  100% NO 
04-13-2015 17:46:45:439
part-r-4.parquet 177.21 MB 32.00 MB  100% NO 
04-13-2015 17:46:44:845
part-r-5.parquet 177.40 MB 32.00 MB  100% NO 
04-13-2015 17:46:44:638
part-r-6.parquet 177.33 MB 32.00 MB  100% NO 
04-13-2015 17:46:44:648

It seems that the API saveAsParquetFile does not distribute/broadcast the 
hadoopconfiguration to executors like the other API such as saveAsTextFile.The 
configutation fs.local.block.size only take effects on Driver.
If I set that configuration before loading parquet files,the problem is gone.
Could anyone help me verify this problem?

Thanks
Zhang Xiongfei



-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: deployment of spark on mesos and data locality in tachyon/hdfs

[Haoyuan Li]

Response inline.

On Tue, Mar 31, 2015 at 10:41 PM, Sean Bigdatafun sean.bigdata...@gmail.com
 wrote:

 (resending...)

 I was thinking the same setup… But the more I think of this problem, and
 the more interesting this could be.

 If we allocate 50% total memory to Tachyon statically, then the Mesos
 benefits of dynamically scheduling resources go away altogether.


People can still benefits from Mesos' dynamically scheduling of the rest
memory as well as compute resource.



 Can Tachyon be resource managed by Mesos (dynamically)? Any thought or
 comment?



This requires some integration work.

Best,

Haoyuan



 Sean





 Hi Haoyuan,

 So on each mesos slave node I should allocate/section off some amount
 of memory for tachyon (let's say 50% of the total memory) and the rest
 for regular mesos tasks?

 This means, on each slave node I would have tachyon worker (+ hdfs
 configuration to talk to s3 or the hdfs datanode) and the mesos slave
 ?process. Is this correct?





 --
 --Sean





-- 
Haoyuan Li
AMPLab, EECS, UC Berkeley
http://www.cs.berkeley.edu/~haoyuan/

Re: deployment of spark on mesos and data locality in tachyon/hdfs

[Sean Bigdatafun]

(resending...)

I was thinking the same setup… But the more I think of this problem, and
the more interesting this could be.

If we allocate 50% total memory to Tachyon statically, then the Mesos
benefits of dynamically scheduling resources go away altogether.

Can Tachyon be resource managed by Mesos (dynamically)? Any thought or
comment?

Sean





 Hi Haoyuan,

 So on each mesos slave node I should allocate/section off some amount
 of memory for tachyon (let's say 50% of the total memory) and the rest
 for regular mesos tasks?

 This means, on each slave node I would have tachyon worker (+ hdfs
 configuration to talk to s3 or the hdfs datanode) and the mesos slave
 ?process. Is this correct?





-- 
--Sean

deployment of spark on mesos and data locality in tachyon/hdfs

[Ankur Chauhan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi,

I am fairly new to the spark ecosystem and I have been trying to setup
a spark on mesos deployment. I can't seem to figure out the best
practices around HDFS and Tachyon. The documentation about Spark's
data-locality section seems to point that each of my mesos slave nodes
should also run a hdfs datanode. This seems fine but I can't seem to
figure out how I would go about pushing tachyon into the mix.

How should i organize my cluster?
Should tachyon be colocated on my mesos worker nodes? or should all
the spark jobs reach out to a separate hdfs/tachyon cluster.

- -- Ankur Chauhan
-BEGIN PGP SIGNATURE-

iQEcBAEBAgAGBQJVGy4bAAoJEOSJAMhvLp3L5bkH/0MECyZkh3ptWzmsNnSNfGWp
Oh93TUfD+foXO2ya9D+hxuyAxbjfXs/68aCWZsUT6qdlBQU9T1vX+CmPOnpY1KPN
NJP3af+VK0osaFPo6k28OTql1iTnvb9Nq+WDlohxBC/hZtoYl4cVxu8JmRlou/nb
/wfpp0ShmJnlxsoPa6mVdwzjUjVQAfEpuet3Ow5veXeA9X7S55k/h0ZQrZtO8eXL
jJsKaT8ne9WZPhZwA4PkdzTxkXF3JNveCIKPzNttsJIaLlvd0nLA/wu6QWmxskp6
iliGSmEk5P1zZWPPnk+TPIqbA0Ttue7PeXpSrbA9+pYiNT4R/wAneMvmpTABuR4=
=8ijP
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: deployment of spark on mesos and data locality in tachyon/hdfs

[Haoyuan Li]

Tachyon should be co-located with Spark in this case.

Best,

Haoyuan

On Tue, Mar 31, 2015 at 4:30 PM, Ankur Chauhan achau...@brightcove.com
wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 Hi,

 I am fairly new to the spark ecosystem and I have been trying to setup
 a spark on mesos deployment. I can't seem to figure out the best
 practices around HDFS and Tachyon. The documentation about Spark's
 data-locality section seems to point that each of my mesos slave nodes
 should also run a hdfs datanode. This seems fine but I can't seem to
 figure out how I would go about pushing tachyon into the mix.

 How should i organize my cluster?
 Should tachyon be colocated on my mesos worker nodes? or should all
 the spark jobs reach out to a separate hdfs/tachyon cluster.

 - -- Ankur Chauhan
 -BEGIN PGP SIGNATURE-

 iQEcBAEBAgAGBQJVGy4bAAoJEOSJAMhvLp3L5bkH/0MECyZkh3ptWzmsNnSNfGWp
 Oh93TUfD+foXO2ya9D+hxuyAxbjfXs/68aCWZsUT6qdlBQU9T1vX+CmPOnpY1KPN
 NJP3af+VK0osaFPo6k28OTql1iTnvb9Nq+WDlohxBC/hZtoYl4cVxu8JmRlou/nb
 /wfpp0ShmJnlxsoPa6mVdwzjUjVQAfEpuet3Ow5veXeA9X7S55k/h0ZQrZtO8eXL
 jJsKaT8ne9WZPhZwA4PkdzTxkXF3JNveCIKPzNttsJIaLlvd0nLA/wu6QWmxskp6
 iliGSmEk5P1zZWPPnk+TPIqbA0Ttue7PeXpSrbA9+pYiNT4R/wAneMvmpTABuR4=
 =8ijP
 -END PGP SIGNATURE-

 -
 To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
 For additional commands, e-mail: user-h...@spark.apache.org




-- 
Haoyuan Li
AMPLab, EECS, UC Berkeley
http://www.cs.berkeley.edu/~haoyuan/

Re: deployment of spark on mesos and data locality in tachyon/hdfs

[Ankur Chauhan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi Haoyuan,

So on each mesos slave node I should allocate/section off some amount
of memory for tachyon (let's say 50% of the total memory) and the rest
for regular mesos tasks?

This means, on each slave node I would have tachyon worker (+ hdfs
configuration to talk to s3 or the hdfs datanode) and the mesos slave
process. Is this correct?

On 31/03/2015 16:43, Haoyuan Li wrote:
 Tachyon should be co-located with Spark in this case.
 
 Best,
 
 Haoyuan
 
 On Tue, Mar 31, 2015 at 4:30 PM, Ankur Chauhan
 achau...@brightcove.com mailto:achau...@brightcove.com wrote:
 
 Hi,
 
 I am fairly new to the spark ecosystem and I have been trying to
 setup a spark on mesos deployment. I can't seem to figure out the
 best practices around HDFS and Tachyon. The documentation about
 Spark's data-locality section seems to point that each of my mesos
 slave nodes should also run a hdfs datanode. This seems fine but I
 can't seem to figure out how I would go about pushing tachyon into
 the mix.
 
 How should i organize my cluster? Should tachyon be colocated on my
 mesos worker nodes? or should all the spark jobs reach out to a
 separate hdfs/tachyon cluster.
 
 -- Ankur Chauhan
 
 -

 
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
 mailto:user-unsubscr...@spark.apache.org For additional commands,
 e-mail: user-h...@spark.apache.org 
 mailto:user-h...@spark.apache.org
 
 
 
 
 -- Haoyuan Li AMPLab, EECS, UC Berkeley 
 http://www.cs.berkeley.edu/~haoyuan/

- -- 
- -- Ankur Chauhan
-BEGIN PGP SIGNATURE-

iQEcBAEBAgAGBQJVGzKUAAoJEOSJAMhvLp3L3W4IAIVYiEKIZbC1a36/KWo94xYB
dvE4VXxF7z5FWmpuaHBEa+U1XWrR4cLVsQhocusOFn+oC7bstdltt3cGNAuwFSv6
Oogs4Sl1J4YZm8omKVdCkwD6Hv71HSntM8llz3qTW+Ljk2aKhfvNtp5nioQAm3e+
bs4ZKlCBij/xV3LbYYIePSS3lL0d9m1qEDJvi6jFcfm3gnBYeNeL9x92B5ylyth0
BGHnPN4sV/yopgrqOimLb12gSexHGNP1y6JBYy8NrHRY8SxkZ4sWKuyDnGDCOPOc
HC14Parf5Ly5FEz5g5WjF6HrXRdPlgr2ABxSLWOAB/siXsX9o/4yCy7NtDNcL6Y=
=f2xI
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: rdd.toDF().saveAsParquetFile(tachyon://host:19998/test)

[Yin Huai]

You are hitting https://issues.apache.org/jira/browse/SPARK-6330. It has
been fixed in 1.3.1, which will be released soon.

On Fri, Mar 27, 2015 at 10:42 PM, sud_self 852677...@qq.com wrote:

 spark version is 1.3.0 with tanhyon-0.6.1

 QUESTION DESCRIPTION: rdd.saveAsObjectFile(tachyon://host:19998/test)
 and
 rdd.saveAsTextFile(tachyon://host:19998/test)  succeed,   but
 rdd.toDF().saveAsParquetFile(tachyon://host:19998/test) failure.

 ERROR MESSAGE:java.lang.IllegalArgumentException: Wrong FS:
 tachyon://host:19998/test, expected: hdfs://host:8020
 at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645)
 at
 org.apache.hadoop.fs.FileSystem.makeQualified(FileSystem.java:465)
 at

 org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$6.apply(newParquet.scala:252)
 at

 org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$6.apply(newParquet.scala:251)



 --
 View this message in context:
 http://apache-spark-user-list.1001560.n3.nabble.com/rdd-toDF-saveAsParquetFile-tachyon-host-19998-test-tp22264.html
 Sent from the Apache Spark User List mailing list archive at Nabble.com.

 -
 To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
 For additional commands, e-mail: user-h...@spark.apache.org

rdd.toDF().saveAsParquetFile(tachyon://host:19998/test)

[sud_self]

spark version is 1.3.0 with tanhyon-0.6.1

QUESTION DESCRIPTION: rdd.saveAsObjectFile(tachyon://host:19998/test)  and 
rdd.saveAsTextFile(tachyon://host:19998/test)  succeed,   but
rdd.toDF().saveAsParquetFile(tachyon://host:19998/test) failure.

ERROR MESSAGE:java.lang.IllegalArgumentException: Wrong FS:
tachyon://host:19998/test, expected: hdfs://host:8020
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645)
at org.apache.hadoop.fs.FileSystem.makeQualified(FileSystem.java:465)
at
org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$6.apply(newParquet.scala:252)
at
org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$6.apply(newParquet.scala:251)



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/rdd-toDF-saveAsParquetFile-tachyon-host-19998-test-tp22264.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: RE: Building spark over specified tachyon

[fightf...@163.com]

Thanks, Jerry
I got that way. Just to make sure whether there can be some option to directly 
specifying tachyon version.




fightf...@163.com
 
From: Shao, Saisai
Date: 2015-03-16 11:10
To: fightf...@163.com
CC: user
Subject: RE: Building spark over specified tachyon
I think you could change the pom file under Spark project to update the Tachyon 
related dependency version and rebuild it again (in case API is compatible, and 
behavior is the same).
 
I’m not sure is there any command you can use to compile against Tachyon 
version.
 
Thanks
Jerry
 
From: fightf...@163.com [mailto:fightf...@163.com] 
Sent: Monday, March 16, 2015 11:01 AM
To: user
Subject: Building spark over specified tachyon
 
Hi, all
Noting that the current spark releases are built-in with tachyon 0.5.0 ,
if we want to recompile spark with maven and targeting on specific tachyon 
version (let's say the most recent 0.6.0 release),
how should that be done? What maven compile command should be like ?
 
Thanks,
Sun. 
 


fightf...@163.com

RE: Building spark over specified tachyon

[Shao, Saisai]

I think you could change the pom file under Spark project to update the Tachyon 
related dependency version and rebuild it again (in case API is compatible, and 
behavior is the same).

I'm not sure is there any command you can use to compile against Tachyon 
version.

Thanks
Jerry

From: fightf...@163.com [mailto:fightf...@163.com]
Sent: Monday, March 16, 2015 11:01 AM
To: user
Subject: Building spark over specified tachyon

Hi, all
Noting that the current spark releases are built-in with tachyon 0.5.0 ,
if we want to recompile spark with maven and targeting on specific tachyon 
version (let's say the most recent 0.6.0 release),
how should that be done? What maven compile command should be like ?

Thanks,
Sun.


fightf...@163.commailto:fightf...@163.com

Building spark over specified tachyon

[fightf...@163.com]

Hi, all
Noting that the current spark releases are built-in with tachyon 0.5.0 ,
if we want to recompile spark with maven and targeting on specific tachyon 
version (let's say the most recent 0.6.0 release),
how should that be done? What maven compile command should be like ?

Thanks,
Sun. 



fightf...@163.com

Re: Re: Building spark over specified tachyon

[fightf...@163.com]

Thanks haoyuan.




fightf...@163.com
 
From: Haoyuan Li
Date: 2015-03-16 12:59
To: fightf...@163.com
CC: Shao, Saisai; user
Subject: Re: RE: Building spark over specified tachyon
Here is a patch: https://github.com/apache/spark/pull/4867

On Sun, Mar 15, 2015 at 8:46 PM, fightf...@163.com fightf...@163.com wrote:
Thanks, Jerry
I got that way. Just to make sure whether there can be some option to directly 
specifying tachyon version.




fightf...@163.com
 
From: Shao, Saisai
Date: 2015-03-16 11:10
To: fightf...@163.com
CC: user
Subject: RE: Building spark over specified tachyon
I think you could change the pom file under Spark project to update the Tachyon 
related dependency version and rebuild it again (in case API is compatible, and 
behavior is the same).
 
I’m not sure is there any command you can use to compile against Tachyon 
version.
 
Thanks
Jerry
 
From: fightf...@163.com [mailto:fightf...@163.com] 
Sent: Monday, March 16, 2015 11:01 AM
To: user
Subject: Building spark over specified tachyon
 
Hi, all
Noting that the current spark releases are built-in with tachyon 0.5.0 ,
if we want to recompile spark with maven and targeting on specific tachyon 
version (let's say the most recent 0.6.0 release),
how should that be done? What maven compile command should be like ?
 
Thanks,
Sun. 
 


fightf...@163.com



-- 
Haoyuan Li
AMPLab, EECS, UC Berkeley
http://www.cs.berkeley.edu/~haoyuan/

Re: A way to share RDD directly using Tachyon?

[Akhil Das]

Did you try something like:

myRDD.saveAsObjectFile(tachyon://localhost:19998/Y)
val newRDD = sc.objectFile[MyObject](tachyon://localhost:19998/Y)


Thanks
Best Regards

On Sun, Mar 8, 2015 at 3:59 PM, Yijie Shen henry.yijies...@gmail.com
wrote:

 Hi,

 I would like to share a RDD in several Spark Applications,
 i.e, create one in application A, publish the ID somewhere and get the RDD
 back directly using ID in Application B.

 I know I can use Tachyon just as a filesystem and
 s.saveAsTextFile(tachyon://localhost:19998/Y”) like this.

 But get a RDD directly from tachyon instead of a file can sometimes avoid
 parsing the same file repeatedly in different Apps, I think.

 What am I supposed to do in order to share RDDs to get a better
 performance?


 —
 Best Regards!
 Yijie Shen

A way to share RDD directly using Tachyon?

[Yijie Shen]

Hi,

I would like to share a RDD in several Spark Applications, 
i.e, create one in application A, publish the ID somewhere and get the RDD back 
directly using ID in Application B.

I know I can use Tachyon just as a filesystem and 
s.saveAsTextFile(tachyon://localhost:19998/Y”) like this.

But get a RDD directly from tachyon instead of a file can sometimes avoid 
parsing the same file repeatedly in different Apps, I think.

What am I supposed to do in order to share RDDs to get a better performance?  


— 
Best Regards!
Yijie Shen

Store the shuffled files in memory using Tachyon

[sara mustafa]

Hi all,

Is it possible to store Spark shuffled files on a distributed memory like
Tachyon instead of spilling them to disk?



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Store-the-shuffled-files-in-memory-using-Tachyon-tp21944.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Spark or Tachyon: capture data lineage

[Sven Krasser]

Agreed with Jerry. Aside from Tachyon, seeing this for general debugging
would be very helpful.

Haoyuan, is that feature you are referring to related to
https://issues.apache.org/jira/browse/SPARK-975?

In the interim, I've found the toDebugString() method useful (but it
renders execution as a tree and not as a more general DAG and therefore
doesn't always capture the flow in the way I'd like to review it). Example:

 a = sc.parallelize(range(1,1000)).map(lambda x: (x, x*x)).filter(lambda
x: x[1]1000)
 b = a.join(a)
 print b.toDebugString()
(16) PythonRDD[19] at RDD at PythonRDD.scala:43
 |   MappedRDD[17] at values at NativeMethodAccessorImpl.java:-2
 |   ShuffledRDD[16] at partitionBy at NativeMethodAccessorImpl.java:-2
 +-(16) PairwiseRDD[15] at RDD at PythonRDD.scala:261
|   PythonRDD[14] at RDD at PythonRDD.scala:43
|   UnionRDD[13] at union at NativeMethodAccessorImpl.java:-2
|   PythonRDD[11] at RDD at PythonRDD.scala:43
|   ParallelCollectionRDD[10] at parallelize at PythonRDD.scala:315
|   PythonRDD[12] at RDD at PythonRDD.scala:43
|   ParallelCollectionRDD[10] at parallelize at PythonRDD.scala:315

Best,
-Sven

On Fri, Jan 2, 2015 at 12:32 PM, Haoyuan Li haoyuan...@gmail.com wrote:

 Jerry,

 Great question. Spark and Tachyon capture lineage information at different
 granularities. We are working on an integration between Spark/Tachyon about
 this. Hope to get it ready to be released soon.

 Best,

 Haoyuan

 On Fri, Jan 2, 2015 at 12:24 PM, Jerry Lam chiling...@gmail.com wrote:

 Hi spark developers,

 I was thinking it would be nice to extract the data lineage information
 from a data processing pipeline. I assume that spark/tachyon keeps this
 information somewhere. For instance, a data processing pipeline uses
 datasource A and B to produce C. C is then used by another process to
 produce D and E. Asumming A, B, C, D, E are stored on disk, It would be so
 useful if there is a way to capture this information when we are using
 spark/tachyon to query this data lineage information. For example, give me
 datasets that produce E. It should give me  a graph like (A and B)-C-E.

 Is this something already possible with spark/tachyon? If not, do you
 think it is possible? Does anyone mind to share their experience in
 capturing the data lineage in a data processing pipeline?

 Best Regards,

 Jerry




 --
 Haoyuan Li
 AMPLab, EECS, UC Berkeley
 http://www.cs.berkeley.edu/~haoyuan/




-- 
http://sites.google.com/site/krasser/?utm_source=sig

Re: Spark or Tachyon: capture data lineage

[Haoyuan Li]

Jerry,

Great question. Spark and Tachyon capture lineage information at different
granularities. We are working on an integration between Spark/Tachyon about
this. Hope to get it ready to be released soon.

Best,

Haoyuan

On Fri, Jan 2, 2015 at 12:24 PM, Jerry Lam chiling...@gmail.com wrote:

 Hi spark developers,

 I was thinking it would be nice to extract the data lineage information
 from a data processing pipeline. I assume that spark/tachyon keeps this
 information somewhere. For instance, a data processing pipeline uses
 datasource A and B to produce C. C is then used by another process to
 produce D and E. Asumming A, B, C, D, E are stored on disk, It would be so
 useful if there is a way to capture this information when we are using
 spark/tachyon to query this data lineage information. For example, give me
 datasets that produce E. It should give me  a graph like (A and B)-C-E.

 Is this something already possible with spark/tachyon? If not, do you
 think it is possible? Does anyone mind to share their experience in
 capturing the data lineage in a data processing pipeline?

 Best Regards,

 Jerry




-- 
Haoyuan Li
AMPLab, EECS, UC Berkeley
http://www.cs.berkeley.edu/~haoyuan/

Spark or Tachyon: capture data lineage

[Jerry Lam]

Hi spark developers,

I was thinking it would be nice to extract the data lineage information
from a data processing pipeline. I assume that spark/tachyon keeps this
information somewhere. For instance, a data processing pipeline uses
datasource A and B to produce C. C is then used by another process to
produce D and E. Asumming A, B, C, D, E are stored on disk, It would be so
useful if there is a way to capture this information when we are using
spark/tachyon to query this data lineage information. For example, give me
datasets that produce E. It should give me  a graph like (A and B)-C-E.

Is this something already possible with spark/tachyon? If not, do you think
it is possible? Does anyone mind to share their experience in capturing the
data lineage in a data processing pipeline?

Best Regards,

Jerry

Re: UpdateStateByKey persist to Tachyon

[amkcom]

bumping this thread up



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/UpdateStateByKey-persist-to-Tachyon-tp20798p20930.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Spark on Tachyon

[Peng Cheng]

IMHO: cache doesn't provide redundancy, and its in the same jvm, so its much
faster.



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-Tachyon-tp1463p20800.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

spark-ec2 starts hdfs1, tachyon but not spark

[Al Thompson]

Hi All:

I am new to Spark. I recently checked out and built spark 1.2 RC2 as an
assembly.
I then ran spark-ec2 according to:

http://spark.apache.org/docs/latest/ec2-scripts.html

I got master and slave instances in EC2 after running

./src/spark/ec2/spark-ec2 -k mykey -i mykey.pem -s 1 launch myclus

All seem to run OK. However, I got no web UI's for spark master or slave.
Logging into the nodes, I see HDFS and Tachyon processes but none for Spark.

The /root/tachyon folder has a full complement of files including conf,
logs and so forth:

$ ls /root/tachyon
bin   docs libexec  logs README.md  target
conf  journal  LICENSE  pom.xml  src

The /root/spark folder only has a conf dir:

$ ls /root/spark
conf

If I try to run the spark setup script I see errors like::

Setting up spark-standalone
RSYNC'ing /root/spark/conf to slaves...
ec2-some-ip.compute-1.amazonaws.com
./spark-standalone/setup.sh: line 22: /root/spark/sbin/stop-all.sh: No such
file or directory
./spark-standalone/setup.sh: line 27: /root/spark/sbin/start-master.sh: No
such file or directory
./spark-standalone/setup.sh: line 33: /root/spark/sbin/start-slaves.sh: No
such file or directory

This makes it seem that something did not get unpacked properly for Spark.
Any hints or workarounds to fixing this?

Cheers,
Al

Re: Persist kafka streams to text file, tachyon error?

[Haoyuan Li]

StorageLevel.OFF_HEAP requires to run Tachyon:
http://spark.apache.org/docs/latest/programming-guide.html

If you don't know if you have tachyon or not, you probably don't :)
http://tachyon-project.org/

For local testing, you can use other persist() solutions without running
Tachyon.

Best,

Haoyuan

On Fri, Nov 21, 2014 at 2:48 PM, Joanne Contact joannenetw...@gmail.com
wrote:

 use the right email list.
 -- Forwarded message --
 From: Joanne Contact joannenetw...@gmail.com
 Date: Fri, Nov 21, 2014 at 2:32 PM
 Subject: Persist kafka streams to text file
 To: u...@spark.incubator.apache.org


 Hello I am trying to read kafka stream to a text file by running spark
 from my IDE (IntelliJ IDEA) . The code is similar as a previous thread on
 persisting stream to a text file.

 I am new to spark or scala. I believe the spark is on local mode as the
 console shows
 14/11/21 14:17:11 INFO spark.SparkContext: Spark configuration:
 spark.app.name=local-mode

  I got the following errors. It is related to Tachyon. But I don't know if
 I have tachyon or not.

 14/11/21 14:17:54 WARN storage.TachyonBlockManager: Attempt 1 to create
 tachyon dir null failed
 java.io.IOException: Failed to connect to master localhost/127.0.0.1:19998
 after 5 attempts
 at tachyon.client.TachyonFS.connect(TachyonFS.java:293)
 at tachyon.client.TachyonFS.getFileId(TachyonFS.java:1011)
 at tachyon.client.TachyonFS.exist(TachyonFS.java:633)
 at
 org.apache.spark.storage.TachyonBlockManager$$anonfun$createTachyonDirs$2.apply(TachyonBlockManager.scala:117)
 at
 org.apache.spark.storage.TachyonBlockManager$$anonfun$createTachyonDirs$2.apply(TachyonBlockManager.scala:106)
 at
 scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
 at
 scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
 at
 scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
 at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
 at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
 at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
 at
 org.apache.spark.storage.TachyonBlockManager.createTachyonDirs(TachyonBlockManager.scala:106)
 at
 org.apache.spark.storage.TachyonBlockManager.init(TachyonBlockManager.scala:57)
 at
 org.apache.spark.storage.BlockManager.tachyonStore$lzycompute(BlockManager.scala:88)
 at
 org.apache.spark.storage.BlockManager.tachyonStore(BlockManager.scala:82)
 at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:729)
 at
 org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:594)
 at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:145)
 at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:70)
 at org.apache.spark.rdd.RDD.iterator(RDD.scala:227)
 at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:87)
 at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
 at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
 at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
 at org.apache.spark.scheduler.Task.run(Task.scala:54)
 at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
 at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:745)
 Caused by: tachyon.org.apache.thrift.TException: Failed to connect to
 master localhost/127.0.0.1:19998 after 5 attempts
 at tachyon.master.MasterClient.connect(MasterClient.java:178)
 at tachyon.client.TachyonFS.connect(TachyonFS.java:290)
 ... 28 more
 Caused by: tachyon.org.apache.thrift.transport.TTransportException:
 java.net.ConnectException: Connection refused
 at tachyon.org.apache.thrift.transport.TSocket.open(TSocket.java:185)
 at
 tachyon.org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
 at tachyon.master.MasterClient.connect(MasterClient.java:156)
 ... 29 more
 Caused by: java.net.ConnectException: Connection refused
 at java.net.PlainSocketImpl.socketConnect(Native Method)
 at
 java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
 at
 java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
 at
 java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
 at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
 at java.net.Socket.connect(Socket.java:579)
 at tachyon.org.apache.thrift.transport.TSocket.open(TSocket.java:180)
 ... 31 more
 14/11/21 14:17:54 ERROR storage.TachyonBlockManager: Failed 10 attempts to
 create tachyon dir in
 /tmp_spark_tachyon/spark-3dbec68b-f5b8-45e1-bb68-370439839d4a/driver

 I looked at the code. It has the following part. Is that a problem?

 .persist(StorageLevel.OFF_HEAP)

 Any advice?

 Thank you!

 J




-- 
Haoyuan Li
AMPLab, EECS, UC Berkeley
http://www.cs.berkeley.edu/~haoyuan/

Persist kafka streams to text file, tachyon error?

[Joanne Contact]

use the right email list.
-- Forwarded message --
From: Joanne Contact joannenetw...@gmail.com
Date: Fri, Nov 21, 2014 at 2:32 PM
Subject: Persist kafka streams to text file
To: u...@spark.incubator.apache.org


Hello I am trying to read kafka stream to a text file by running spark from
my IDE (IntelliJ IDEA) . The code is similar as a previous thread on
persisting stream to a text file.

I am new to spark or scala. I believe the spark is on local mode as the
console shows
14/11/21 14:17:11 INFO spark.SparkContext: Spark configuration:
spark.app.name=local-mode

 I got the following errors. It is related to Tachyon. But I don't know if
I have tachyon or not.

14/11/21 14:17:54 WARN storage.TachyonBlockManager: Attempt 1 to create
tachyon dir null failed
java.io.IOException: Failed to connect to master localhost/127.0.0.1:19998
after 5 attempts
at tachyon.client.TachyonFS.connect(TachyonFS.java:293)
at tachyon.client.TachyonFS.getFileId(TachyonFS.java:1011)
at tachyon.client.TachyonFS.exist(TachyonFS.java:633)
at
org.apache.spark.storage.TachyonBlockManager$$anonfun$createTachyonDirs$2.apply(TachyonBlockManager.scala:117)
at
org.apache.spark.storage.TachyonBlockManager$$anonfun$createTachyonDirs$2.apply(TachyonBlockManager.scala:106)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
at
org.apache.spark.storage.TachyonBlockManager.createTachyonDirs(TachyonBlockManager.scala:106)
at
org.apache.spark.storage.TachyonBlockManager.init(TachyonBlockManager.scala:57)
at
org.apache.spark.storage.BlockManager.tachyonStore$lzycompute(BlockManager.scala:88)
at org.apache.spark.storage.BlockManager.tachyonStore(BlockManager.scala:82)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:729)
at org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:594)
at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:145)
at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:70)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:227)
at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:87)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
at org.apache.spark.scheduler.Task.run(Task.scala:54)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: tachyon.org.apache.thrift.TException: Failed to connect to
master localhost/127.0.0.1:19998 after 5 attempts
at tachyon.master.MasterClient.connect(MasterClient.java:178)
at tachyon.client.TachyonFS.connect(TachyonFS.java:290)
... 28 more
Caused by: tachyon.org.apache.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused
at tachyon.org.apache.thrift.transport.TSocket.open(TSocket.java:185)
at
tachyon.org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
at tachyon.master.MasterClient.connect(MasterClient.java:156)
... 29 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at tachyon.org.apache.thrift.transport.TSocket.open(TSocket.java:180)
... 31 more
14/11/21 14:17:54 ERROR storage.TachyonBlockManager: Failed 10 attempts to
create tachyon dir in
/tmp_spark_tachyon/spark-3dbec68b-f5b8-45e1-bb68-370439839d4a/driver

I looked at the code. It has the following part. Is that a problem?

.persist(StorageLevel.OFF_HEAP)

Any advice?

Thank you!

J

Storing shuffle files on a Tachyon

[Soumya Simanta]

Is it possible to store spark shuffle files on Tachyon ?

Re: spark-ec2 script with Tachyon

[mrm]

Hi,

Did you manage to figure this out? I would appreciate if you could share the
answer.



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/spark-ec2-script-with-Tachyon-tp9996p15249.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

[Tachyon] Error reading from Parquet files in HDFS

[Evan Chan]

Spark 1.0.2, Tachyon 0.4.1, Hadoop 1.0  (standard EC2 config)

scala val gdeltT =
sqlContext.parquetFile(tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005/)
14/08/21 19:07:14 INFO :
initialize(tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005,
Configuration: core-default.xml, core-site.xml, mapred-default.xml,
mapred-site.xml, hdfs-default.xml, hdfs-site.xml). Connecting to
Tachyon: tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005
14/08/21 19:07:14 INFO : Trying to connect master @ /172.31.42.40:19998
14/08/21 19:07:14 INFO : User registered at the master
ip-172-31-42-40.us-west-2.compute.internal/172.31.42.40:19998 got
UserId 14
14/08/21 19:07:14 INFO : Trying to get local worker host :
ip-172-31-42-40.us-west-2.compute.internal
14/08/21 19:07:14 INFO : No local worker on
ip-172-31-42-40.us-west-2.compute.internal
14/08/21 19:07:14 INFO : Connecting remote worker @
ip-172-31-47-74/172.31.47.74:29998
14/08/21 19:07:14 INFO : tachyon://172.31.42.40:19998
tachyon://172.31.42.40:19998
hdfs://ec2-54-213-113-173.us-west-2.compute.amazonaws.com:9000
14/08/21 19:07:14 INFO :
getFileStatus(tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005):
HDFS Path: 
hdfs://ec2-54-213-113-173.us-west-2.compute.amazonaws.com:9000/gdelt-parquet/1979-2005
TPath: tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005
14/08/21 19:07:14 INFO : tachyon.client.TachyonFS@4b05b3ff
hdfs://ec2-54-213-113-173.us-west-2.compute.amazonaws.com:9000
/gdelt-parquet/1979-2005 tachyon.PrefixList@636c50d3
14/08/21 19:07:14 WARN : tachyon.home is not set. Using
/mnt/tachyon_default_home as the default value.
14/08/21 19:07:14 INFO : Get: /gdelt-parquet/1979-2005/_SUCCESS
14/08/21 19:07:14 INFO : Get: /gdelt-parquet/1979-2005/_metadata
14/08/21 19:07:14 INFO : Get: /gdelt-parquet/1979-2005/part-r-1.parquet



14/08/21 19:07:14 INFO :
getFileStatus(tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005/_metadata):
HDFS Path: 
hdfs://ec2-54-213-113-173.us-west-2.compute.amazonaws.com:9000/gdelt-parquet/1979-2005/_metadata
TPath: tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005/_metadata
14/08/21 19:07:14 INFO :
open(tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005/_metadata,
65536)
14/08/21 19:07:14 ERROR : The machine does not have any local worker.
14/08/21 19:07:14 ERROR : Reading from HDFS directly
14/08/21 19:07:14 ERROR : Reading from HDFS directly
java.io.IOException: can not read class parquet.format.FileMetaData: null
at parquet.format.Util.read(Util.java:50)
at parquet.format.Util.readFileMetaData(Util.java:34)
at 
parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:310)
at parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:296)


I'm not sure why this is saying that, as the Tachyon UI reports all 8
nodes being up?

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: [Tachyon] Error reading from Parquet files in HDFS

[Evan Chan]

The underFS is HDFS btw.

On Thu, Aug 21, 2014 at 12:22 PM, Evan Chan velvia.git...@gmail.com wrote:
 Spark 1.0.2, Tachyon 0.4.1, Hadoop 1.0  (standard EC2 config)

 scala val gdeltT =
 sqlContext.parquetFile(tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005/)
 14/08/21 19:07:14 INFO :
 initialize(tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005,
 Configuration: core-default.xml, core-site.xml, mapred-default.xml,
 mapred-site.xml, hdfs-default.xml, hdfs-site.xml). Connecting to
 Tachyon: tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005
 14/08/21 19:07:14 INFO : Trying to connect master @ /172.31.42.40:19998
 14/08/21 19:07:14 INFO : User registered at the master
 ip-172-31-42-40.us-west-2.compute.internal/172.31.42.40:19998 got
 UserId 14
 14/08/21 19:07:14 INFO : Trying to get local worker host :
 ip-172-31-42-40.us-west-2.compute.internal
 14/08/21 19:07:14 INFO : No local worker on
 ip-172-31-42-40.us-west-2.compute.internal
 14/08/21 19:07:14 INFO : Connecting remote worker @
 ip-172-31-47-74/172.31.47.74:29998
 14/08/21 19:07:14 INFO : tachyon://172.31.42.40:19998
 tachyon://172.31.42.40:19998
 hdfs://ec2-54-213-113-173.us-west-2.compute.amazonaws.com:9000
 14/08/21 19:07:14 INFO :
 getFileStatus(tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005):
 HDFS Path: 
 hdfs://ec2-54-213-113-173.us-west-2.compute.amazonaws.com:9000/gdelt-parquet/1979-2005
 TPath: tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005
 14/08/21 19:07:14 INFO : tachyon.client.TachyonFS@4b05b3ff
 hdfs://ec2-54-213-113-173.us-west-2.compute.amazonaws.com:9000
 /gdelt-parquet/1979-2005 tachyon.PrefixList@636c50d3
 14/08/21 19:07:14 WARN : tachyon.home is not set. Using
 /mnt/tachyon_default_home as the default value.
 14/08/21 19:07:14 INFO : Get: /gdelt-parquet/1979-2005/_SUCCESS
 14/08/21 19:07:14 INFO : Get: /gdelt-parquet/1979-2005/_metadata
 14/08/21 19:07:14 INFO : Get: /gdelt-parquet/1979-2005/part-r-1.parquet

 

 14/08/21 19:07:14 INFO :
 getFileStatus(tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005/_metadata):
 HDFS Path: 
 hdfs://ec2-54-213-113-173.us-west-2.compute.amazonaws.com:9000/gdelt-parquet/1979-2005/_metadata
 TPath: tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005/_metadata
 14/08/21 19:07:14 INFO :
 open(tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005/_metadata,
 65536)
 14/08/21 19:07:14 ERROR : The machine does not have any local worker.
 14/08/21 19:07:14 ERROR : Reading from HDFS directly
 14/08/21 19:07:14 ERROR : Reading from HDFS directly
 java.io.IOException: can not read class parquet.format.FileMetaData: null
 at parquet.format.Util.read(Util.java:50)
 at parquet.format.Util.readFileMetaData(Util.java:34)
 at 
 parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:310)
 at parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:296)


 I'm not sure why this is saying that, as the Tachyon UI reports all 8
 nodes being up?

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: [Tachyon] Error reading from Parquet files in HDFS

[Evan Chan]

And it worked earlier with non-parquet directory.

On Thu, Aug 21, 2014 at 12:22 PM, Evan Chan velvia.git...@gmail.com wrote:
 The underFS is HDFS btw.

 On Thu, Aug 21, 2014 at 12:22 PM, Evan Chan velvia.git...@gmail.com wrote:
 Spark 1.0.2, Tachyon 0.4.1, Hadoop 1.0  (standard EC2 config)

 scala val gdeltT =
 sqlContext.parquetFile(tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005/)
 14/08/21 19:07:14 INFO :
 initialize(tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005,
 Configuration: core-default.xml, core-site.xml, mapred-default.xml,
 mapred-site.xml, hdfs-default.xml, hdfs-site.xml). Connecting to
 Tachyon: tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005
 14/08/21 19:07:14 INFO : Trying to connect master @ /172.31.42.40:19998
 14/08/21 19:07:14 INFO : User registered at the master
 ip-172-31-42-40.us-west-2.compute.internal/172.31.42.40:19998 got
 UserId 14
 14/08/21 19:07:14 INFO : Trying to get local worker host :
 ip-172-31-42-40.us-west-2.compute.internal
 14/08/21 19:07:14 INFO : No local worker on
 ip-172-31-42-40.us-west-2.compute.internal
 14/08/21 19:07:14 INFO : Connecting remote worker @
 ip-172-31-47-74/172.31.47.74:29998
 14/08/21 19:07:14 INFO : tachyon://172.31.42.40:19998
 tachyon://172.31.42.40:19998
 hdfs://ec2-54-213-113-173.us-west-2.compute.amazonaws.com:9000
 14/08/21 19:07:14 INFO :
 getFileStatus(tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005):
 HDFS Path: 
 hdfs://ec2-54-213-113-173.us-west-2.compute.amazonaws.com:9000/gdelt-parquet/1979-2005
 TPath: tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005
 14/08/21 19:07:14 INFO : tachyon.client.TachyonFS@4b05b3ff
 hdfs://ec2-54-213-113-173.us-west-2.compute.amazonaws.com:9000
 /gdelt-parquet/1979-2005 tachyon.PrefixList@636c50d3
 14/08/21 19:07:14 WARN : tachyon.home is not set. Using
 /mnt/tachyon_default_home as the default value.
 14/08/21 19:07:14 INFO : Get: /gdelt-parquet/1979-2005/_SUCCESS
 14/08/21 19:07:14 INFO : Get: /gdelt-parquet/1979-2005/_metadata
 14/08/21 19:07:14 INFO : Get: /gdelt-parquet/1979-2005/part-r-1.parquet

 

 14/08/21 19:07:14 INFO :
 getFileStatus(tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005/_metadata):
 HDFS Path: 
 hdfs://ec2-54-213-113-173.us-west-2.compute.amazonaws.com:9000/gdelt-parquet/1979-2005/_metadata
 TPath: tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005/_metadata
 14/08/21 19:07:14 INFO :
 open(tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005/_metadata,
 65536)
 14/08/21 19:07:14 ERROR : The machine does not have any local worker.
 14/08/21 19:07:14 ERROR : Reading from HDFS directly
 14/08/21 19:07:14 ERROR : Reading from HDFS directly
 java.io.IOException: can not read class parquet.format.FileMetaData: null
 at parquet.format.Util.read(Util.java:50)
 at parquet.format.Util.readFileMetaData(Util.java:34)
 at 
 parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:310)
 at parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:296)


 I'm not sure why this is saying that, as the Tachyon UI reports all 8
 nodes being up?

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

First Bay Area Tachyon meetup: August 25th, hosted by Yahoo! (Limited Space)

[Haoyuan Li]

Hi folks,

We've posted the first Tachyon meetup, which will be on August 25th and is
hosted by Yahoo! (Limited Space):
http://www.meetup.com/Tachyon/events/200387252/ . Hope to see you there!

Best,

Haoyuan

-- 
Haoyuan Li
AMPLab, EECS, UC Berkeley
http://www.cs.berkeley.edu/~haoyuan/

Re: First Bay Area Tachyon meetup: August 25th, hosted by Yahoo! (Limited Space)

[Christopher Nguyen]

Fantastic!

Sent while mobile. Pls excuse typos etc.
On Aug 19, 2014 4:09 PM, Haoyuan Li haoyuan...@gmail.com wrote:

 Hi folks,

 We've posted the first Tachyon meetup, which will be on August 25th and is
 hosted by Yahoo! (Limited Space):
 http://www.meetup.com/Tachyon/events/200387252/ . Hope to see you there!

 Best,

 Haoyuan

 --
 Haoyuan Li
 AMPLab, EECS, UC Berkeley
 http://www.cs.berkeley.edu/~haoyuan/

Re: share/reuse off-heap persisted (tachyon) RDD in SparkContext or saveAsParquetFile on tachyon in SQLContext

[chutium]

spark.speculation was not set, any speculative execution on tachyon side?

tachyon-env.sh only changed following

export TACHYON_MASTER_ADDRESS=test01.zala
#export TACHYON_UNDERFS_ADDRESS=$TACHYON_HOME/underfs
export TACHYON_UNDERFS_ADDRESS=hdfs://test01.zala:8020
export TACHYON_WORKER_MEMORY_SIZE=16GB

test01.zala is master node for HDFS, tachyon, Spark, etc.
worker nodes are
test02.zala
test03.zala
test04.zala

spark-shell run on test02

after parquetFile.saveAsParquetFile(tachyon://test01.zala:19998/parquet_1)

i got FailedToCheckpointException with Failed to rename

but tfs lsr, there are some temporary files and metadata file

0.00 B08-11-2014 16:19:28:054 /parquet_1
881.00 B  08-11-2014 16:19:28:054  In Memory  /parquet_1/_metadata
0.00 B08-11-2014 16:19:28:314 /parquet_1/_temporary
0.00 B08-11-2014 16:19:28:314 /parquet_1/_temporary/0
0.00 B08-11-2014 16:19:28:931
/parquet_1/_temporary/0/_temporary
0.00 B08-11-2014 16:19:28:931
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_00_33
0.00 B08-11-2014 16:19:28:931  In Memory 
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_00_33/part-r-1.parquet
0.00 B08-11-2014 16:19:28:940
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_01_35
0.00 B08-11-2014 16:19:28:940  In Memory 
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_01_35/part-r-2.parquet
0.00 B08-11-2014 16:19:28:962
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_03_36
0.00 B08-11-2014 16:19:28:962  In Memory 
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_03_36/part-r-4.parquet
0.00 B08-11-2014 16:19:28:971
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_02_34
0.00 B08-11-2014 16:19:28:971  In Memory 
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_02_34/part-r-3.parquet
0.00 B08-11-2014 16:20:06:349
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_01_37
0.00 B08-11-2014 16:20:06:349  In Memory 
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_01_37/part-r-2.parquet
0.00 B08-11-2014 16:20:09:519
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_03_38
0.00 B08-11-2014 16:20:09:519  In Memory 
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_03_38/part-r-4.parquet
0.00 B08-11-2014 16:20:18:777
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_02_39
0.00 B08-11-2014 16:20:18:777  In Memory 
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_02_39/part-r-3.parquet
0.00 B08-11-2014 16:20:28:315
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_00_40
0.00 B08-11-2014 16:20:28:315  In Memory 
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_00_40/part-r-1.parquet
0.00 B08-11-2014 16:20:38:382
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_01_41
0.00 B08-11-2014 16:20:38:382  In Memory 
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_01_41/part-r-2.parquet
0.00 B08-11-2014 16:20:40:681
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_03_42
0.00 B08-11-2014 16:20:40:681  In Memory 
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_03_42/part-r-4.parquet
0.00 B08-11-2014 16:20:50:376
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_02_43
0.00 B08-11-2014 16:20:50:376  In Memory 
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_02_43/part-r-3.parquet
0.00 B08-11-2014 16:21:00:932
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_00_44
0.00 B08-11-2014 16:21:00:932  In Memory 
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_00_44/part-r-1.parquet
0.00 B08-11-2014 16:21:10:355
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_01_45
0.00 B08-11-2014 16:21:10:355  In Memory 
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_01_45/part-r-2.parquet
0.00 B08-11-2014 16:21:11:468
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_03_46
0.00 B08-11-2014 16:21:11:468  In Memory 
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_03_46/part-r-4.parquet
0.00 B08-11-2014 16:21:21:681
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_02_47
0.00 B08-11-2014 16:21:21:681  In Memory 
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_02_47/part-r-3.parquet
0.00 B08-11-2014 16:21:32:583

Re: share/reuse off-heap persisted (tachyon) RDD in SparkContext or saveAsParquetFile on tachyon in SQLContext

[chutium]

more interesting is if spark-shell started on master node (test01)

then

parquetFile.saveAsParquetFile(tachyon://test01.zala:19998/parquet_tablex)

14/08/12 11:42:06 INFO : initialize(tachyon://...
...
...
14/08/12 11:42:06 INFO : File does not exist:
tachyon://test01.zala:19998/parquet_tablex/_metadata
14/08/12 11:42:06 INFO : getWorkingDirectory: /
14/08/12 11:42:06 INFO :
create(tachyon://test01.zala:19998/parquet_tablex/_metadata, rw-r--r--,
true, 65536, 1, 33554432, null)
14/08/12 11:42:06 WARN : tachyon.home is not set. Using
/mnt/tachyon_default_home as the default value.
14/08/12 11:42:06 INFO : Trying to get local worker host : test01.zala
14/08/12 11:42:06 ERROR : No local worker on test01.zala
NoWorkerException(message:No local worker on test01.zala)
at
tachyon.thrift.MasterService$user_getWorker_result$user_getWorker_resultStandardScheme.read(MasterService.java:25675)
at
tachyon.thrift.MasterService$user_getWorker_result$user_getWorker_resultStandardScheme.read(MasterService.java:25652)
at
tachyon.thrift.MasterService$user_getWorker_result.read(MasterService.java:25591)
at
tachyon.org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at
tachyon.thrift.MasterService$Client.recv_user_getWorker(MasterService.java:832)
at
tachyon.thrift.MasterService$Client.user_getWorker(MasterService.java:818)
at tachyon.master.MasterClient.user_getWorker(MasterClient.java:648)
at tachyon.worker.WorkerClient.connect(WorkerClient.java:199)
at tachyon.worker.WorkerClient.mustConnect(WorkerClient.java:360)
at
tachyon.worker.WorkerClient.getUserUfsTempFolder(WorkerClient.java:298)
at
tachyon.client.TachyonFS.createAndGetUserUfsTempFolder(TachyonFS.java:270)
at tachyon.client.FileOutStream.init(FileOutStream.java:72)
at tachyon.client.TachyonFile.getOutStream(TachyonFile.java:207)
at tachyon.hadoop.AbstractTFS.create(AbstractTFS.java:102)
at tachyon.hadoop.TFS.create(TFS.java:24)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:887)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:784)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:773)
at
parquet.hadoop.ParquetFileWriter.writeMetadataFile(ParquetFileWriter.java:344)
at
org.apache.spark.sql.parquet.ParquetTypesConverter$.writeMetaData(ParquetTypes.scala:345)
at
org.apache.spark.sql.parquet.ParquetRelation$.createEmpty(ParquetRelation.scala:142)
at
org.apache.spark.sql.parquet.ParquetRelation$.create(ParquetRelation.scala:120)
at
org.apache.spark.sql.execution.SparkStrategies$ParquetOperations$.apply(SparkStrategies.scala:197)
at
org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58)
at
org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at
org.apache.spark.sql.catalyst.planning.QueryPlanner.apply(QueryPlanner.scala:59)
at
org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan$lzycompute(SQLContext.scala:399)
at
org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan(SQLContext.scala:397)
at
org.apache.spark.sql.SQLContext$QueryExecution.executedPlan$lzycompute(SQLContext.scala:403)
at
org.apache.spark.sql.SQLContext$QueryExecution.executedPlan(SQLContext.scala:403)
at
org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:406)
at
org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:406)
at
org.apache.spark.sql.SchemaRDDLike$class.saveAsParquetFile(SchemaRDDLike.scala:77)
at
org.apache.spark.sql.SchemaRDD.saveAsParquetFile(SchemaRDD.scala:103)
at $line12.$read$$iwC$$iwC$$iwC$$iwC.init(console:17)
at $line12.$read$$iwC$$iwC$$iwC.init(console:22)
at $line12.$read$$iwC$$iwC.init(console:24)
at $line12.$read$$iwC.init(console:26)
at $line12.$read.init(console:28)
at $line12.$read$.init(console:32)
at $line12.$read$.clinit(console)
at $line12.$eval$.init(console:7)
at $line12.$eval$.clinit(console)
at $line12.$eval.$print(console)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:789)
at
org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1062)
at
org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:615

share/reuse off-heap persisted (tachyon) RDD in SparkContext or saveAsParquetFile on tachyon in SQLContext

[chutium]

sharing /reusing RDDs is always useful for many use cases, is this possible
via persisting RDD on tachyon?

such as off heap persist a named RDD into a given path (instead of
/tmp_spark_tachyon/spark-xxx-xxx-xxx)
or
saveAsParquetFile on tachyon

i tried to save a SchemaRDD on tachyon, 

val parquetFile =
sqlContext.parquetFile(hdfs://test01.zala:8020/user/hive/warehouse/parquet_tables.db/some_table/)
parquetFile.saveAsParquetFile(tachyon://test01.zala:19998/parquet_1)

but always error, first error message is:

14/08/11 16:19:28 INFO storage.BlockManagerInfo: Added broadcast_6_piece0 in
memory on test03.zala:37377 (size: 18.7 KB, free: 16.6 GB)
14/08/11 16:20:06 WARN scheduler.TaskSetManager: Lost task 1.0 in stage 3.0
(TID 35, test04.zala): java.io.IOException:
FailedToCheckpointException(message:Failed to rename
hdfs://test01.zala:8020/tmp/tachyon/workers/140776003/31806/730 to
hdfs://test01.zala:8020/tmp/tachyon/data/730)
tachyon.worker.WorkerClient.addCheckpoint(WorkerClient.java:112)
tachyon.client.TachyonFS.addCheckpoint(TachyonFS.java:168)
tachyon.client.FileOutStream.close(FileOutStream.java:104)
   
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70)
   
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:103)
parquet.hadoop.ParquetFileWriter.end(ParquetFileWriter.java:321)
   
parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:111)
   
parquet.hadoop.ParquetRecordWriter.close(ParquetRecordWriter.java:73)
   
org.apache.spark.sql.parquet.InsertIntoParquetTable.org$apache$spark$sql$parquet$InsertIntoParquetTable$$writeShard$1(ParquetTableOperations.scala:259)
   
org.apache.spark.sql.parquet.InsertIntoParquetTable$$anonfun$saveAsHadoopFile$1.apply(ParquetTableOperations.scala:272)
   
org.apache.spark.sql.parquet.InsertIntoParquetTable$$anonfun$saveAsHadoopFile$1.apply(ParquetTableOperations.scala:272)
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
org.apache.spark.scheduler.Task.run(Task.scala:54)
   
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:199)
   
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:722)



hdfs://test01.zala:8020/tmp/tachyon/
already chmod to 777, both owner and group is same as spark/tachyon startup
user

off-heap persist or saveAs normal text file on tachyon works fine.

CDH 5.1.0, spark 1.1.0 snapshot, tachyon 0.6 snapshot



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/share-reuse-off-heap-persisted-tachyon-RDD-in-SparkContext-or-saveAsParquetFile-on-tachyon-in-SQLCont-tp11897.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: share/reuse off-heap persisted (tachyon) RDD in SparkContext or saveAsParquetFile on tachyon in SQLContext

[Haoyuan Li]

Is the speculative execution enabled?

Best,

Haoyuan


On Mon, Aug 11, 2014 at 8:08 AM, chutium teng@gmail.com wrote:

 sharing /reusing RDDs is always useful for many use cases, is this possible
 via persisting RDD on tachyon?

 such as off heap persist a named RDD into a given path (instead of
 /tmp_spark_tachyon/spark-xxx-xxx-xxx)
 or
 saveAsParquetFile on tachyon

 i tried to save a SchemaRDD on tachyon,

 val parquetFile =

 sqlContext.parquetFile(hdfs://test01.zala:8020/user/hive/warehouse/parquet_tables.db/some_table/)
 parquetFile.saveAsParquetFile(tachyon://test01.zala:19998/parquet_1)

 but always error, first error message is:

 14/08/11 16:19:28 INFO storage.BlockManagerInfo: Added broadcast_6_piece0
 in
 memory on test03.zala:37377 (size: 18.7 KB, free: 16.6 GB)
 14/08/11 16:20:06 WARN scheduler.TaskSetManager: Lost task 1.0 in stage 3.0
 (TID 35, test04.zala): java.io.IOException:
 FailedToCheckpointException(message:Failed to rename
 hdfs://test01.zala:8020/tmp/tachyon/workers/140776003/31806/730 to
 hdfs://test01.zala:8020/tmp/tachyon/data/730)
 tachyon.worker.WorkerClient.addCheckpoint(WorkerClient.java:112)
 tachyon.client.TachyonFS.addCheckpoint(TachyonFS.java:168)
 tachyon.client.FileOutStream.close(FileOutStream.java:104)


 org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70)

 org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:103)
 parquet.hadoop.ParquetFileWriter.end(ParquetFileWriter.java:321)


 parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:111)

 parquet.hadoop.ParquetRecordWriter.close(ParquetRecordWriter.java:73)

 org.apache.spark.sql.parquet.InsertIntoParquetTable.org
 $apache$spark$sql$parquet$InsertIntoParquetTable$$writeShard$1(ParquetTableOperations.scala:259)


 org.apache.spark.sql.parquet.InsertIntoParquetTable$$anonfun$saveAsHadoopFile$1.apply(ParquetTableOperations.scala:272)


 org.apache.spark.sql.parquet.InsertIntoParquetTable$$anonfun$saveAsHadoopFile$1.apply(ParquetTableOperations.scala:272)
 org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
 org.apache.spark.scheduler.Task.run(Task.scala:54)

 org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:199)


 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)


 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 java.lang.Thread.run(Thread.java:722)



 hdfs://test01.zala:8020/tmp/tachyon/
 already chmod to 777, both owner and group is same as spark/tachyon startup
 user

 off-heap persist or saveAs normal text file on tachyon works fine.

 CDH 5.1.0, spark 1.1.0 snapshot, tachyon 0.6 snapshot



 --
 View this message in context:
 http://apache-spark-user-list.1001560.n3.nabble.com/share-reuse-off-heap-persisted-tachyon-RDD-in-SparkContext-or-saveAsParquetFile-on-tachyon-in-SQLCont-tp11897.html
 Sent from the Apache Spark User List mailing list archive at Nabble.com.

 -
 To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
 For additional commands, e-mail: user-h...@spark.apache.org




-- 
Haoyuan Li
AMPLab, EECS, UC Berkeley
http://www.cs.berkeley.edu/~haoyuan/

Re: Spark-sql with Tachyon cache

[Michael Armbrust]

We are investigating various ways to integrate with Tachyon.  I'll note
that you can already use saveAsParquetFile and
parquetFile(...).registerAsTable(tableName) (soon to be registerTempTable
in Spark 1.1) to store data into tachyon and query it with Spark SQL.


On Fri, Aug 1, 2014 at 1:42 AM, Dariusz Kobylarz darek.kobyl...@gmail.com
wrote:

  Hi, I would like to ask if spark-sql tables cached by Tachyon is a
 feature to be migrated from shark.

 I imagine from the user perspective it would look like this:

 CREATE TABLE data TBLPROPERTIES(sparksql.cache = tachyon) AS SELECT a, b, 
 c from data_on_disk WHERE month=May;

spark-ec2 script with Tachyon

[nit]

Hi,

It seems that spark-ec2 script deploys Tachyon module along with other
setup.
I am trying to use .persist(OFF_HEAP) for RDD persistence, but on worker I
see this error
--
 Failed to connect (2) to master localhost/127.0.0.1:19998 :
java.net.ConnectException: Connection refused
--

From netstat I see that worker is connected to master node on port 19998
--
Proto Recv-Q Send-Q Local Address   Foreign Address
State 
tcp0  0 ip-10-16-132-190.ec2.:49239 ip-10-158-45-248.ec2.:19998
ESTABLISHED 
--

Does Tachyon on EC work out of the box? or does it requite further
configuration ?

Am I supposed to set  spark.tachyonStore.url to Masters IP ?



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/spark-ec2-script-with-Tachyon-tp9996.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Cannot create dir in Tachyon when running Spark with OFF_HEAP caching (FileDoesNotExistException)

[Teng Long]

More updates:

Seems in TachyonBlockManager.scala(line 118) of Spark 1.1.0, the
TachyonFS.mkdir() method is called, which creates a directory in Tachyon.
Right after that, TachyonFS.getFile() method is called. In all the versions
of Tachyon I tried (0.4.1, 0.4.0), the second method will return a null on
directory paths. Is it just me? Anyone else using Spark with Tachyon had any
problems?

Thanks!



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Cannot-create-dir-in-Tachyon-when-running-Spark-with-OFF-HEAP-caching-FileDoesNotExistException-tp8959p9110.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Cannot create dir in Tachyon when running Spark with OFF_HEAP caching (FileDoesNotExistException)

[Teng Long]

Hi guys,

I'm running Spark 1.0.0 with Tachyon 0.4.1, both in single node mode.
Tachyon's own tests (./bin/tachyon runTests) works good, and manual file
system operation like mkdir works well. But when I tried to run a very
simple Spark task with RDD persist as OFF_HEAP, I got the following
FileDoesNotExistException error.

My platform is Ubuntu12.04 x64.

More information:

The Spark task is simply in the interaction mode:
-
import org.apache.spark.storage.StorageLevel
val tf = sc.textFile(README.md) // the file is there in the directory.
tf.persist(StorageLevel.OFF_HEAP)
tf.count()
--
I tried other Storage levels like MEM_ONLY, MEM_AND_DISK, etc, and they all
worked fine.

The same error happened on both Tachyon 0.4.1 and Tachyon 0.4.1-thrifty, for
both binary versions or build-from-src versions.

Even more information:

I further track down the Connecting local worker @ right before the
Exception, and find the problem might be in the tachyon.client.connect()
method. 

Did any of you guys have this problem? If yes, how did you solve it?

Thanks!

-
Exception output:
-


14/07/07 14:03:47 INFO : Trying to connect master @
datanode6/10.10.10.46:19998
14/07/07 14:03:47 INFO : User registered at the master
datanode6/10.10.10.46:19998 got UserId 21
14/07/07 14:03:47 INFO : Trying to get local worker host : datanode6
14/07/07 14:03:47 INFO : Connecting local worker @
datanode6.ssi.samsung.com/10.10.10.46:29998
14/07/07 14:03:47 INFO :
FileDoesNotExistException(message://spark-704334db-270a-48c5-ac52-e646f1ea1aa0/driver/spark-tachyon-20140707140347-6c0a)//spark-704334db-270a-48c5-ac52-e646f1ea1aa0/driver/spark-tachyon-20140707140347-6c0a
14/07/07 14:03:47 INFO storage.TachyonBlockManager: Created tachyon
directory at null
14/07/07 14:03:47 WARN storage.BlockManager: Putting block rdd_1_0 failed
14/07/07 14:03:47 ERROR executor.Executor: Exception in task ID 0
java.lang.NullPointerException
at 
org.apache.spark.util.Utils$.registerShutdownDeleteDir(Utils.scala:196)
at
org.apache.spark.storage.TachyonBlockManager$$anonfun$addShutdownHook$1.apply(TachyonBlockManager.scala:137)
at
org.apache.spark.storage.TachyonBlockManager$$anonfun$addShutdownHook$1.apply(TachyonBlockManager.scala:137)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
at
org.apache.spark.storage.TachyonBlockManager.addShutdownHook(TachyonBlockManager.scala:137)
at
org.apache.spark.storage.TachyonBlockManager.init(TachyonBlockManager.scala:60)
at
org.apache.spark.storage.BlockManager.tachyonStore$lzycompute(BlockManager.scala:69)
at
org.apache.spark.storage.BlockManager.tachyonStore(BlockManager.scala:64)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:681)
at org.apache.spark.storage.BlockManager.put(BlockManager.scala:574)
at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:108)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:227)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
at org.apache.spark.scheduler.Task.run(Task.scala:51)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
14/07/07 14:03:47 WARN scheduler.TaskSetManager: Lost TID 0 (task 0.0:0)
14/07/07 14:03:47 WARN scheduler.TaskSetManager: Loss was due to
java.lang.NullPointerException
java.lang.NullPointerException
at 
org.apache.spark.util.Utils$.registerShutdownDeleteDir(Utils.scala:196)
at
org.apache.spark.storage.TachyonBlockManager$$anonfun$addShutdownHook$1.apply(TachyonBlockManager.scala:137)
at
org.apache.spark.storage.TachyonBlockManager$$anonfun$addShutdownHook$1.apply(TachyonBlockManager.scala:137)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
at
org.apache.spark.storage.TachyonBlockManager.addShutdownHook(TachyonBlockManager.scala:137)
at
org.apache.spark.storage.TachyonBlockManager.init(TachyonBlockManager.scala:60)
at
org.apache.spark.storage.BlockManager.tachyonStore$lzycompute(BlockManager.scala:69)
at
org.apache.spark.storage.BlockManager.tachyonStore(BlockManager.scala:64)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:681