Re: Cannot create dir in Tachyon when running Spark with OFF_HEAP caching (FileDoesNotExistException)

2014-07-08 Thread Teng Long
More updates: Seems in TachyonBlockManager.scala(line 118) of Spark 1.1.0, the TachyonFS.mkdir() method is called, which creates a directory in Tachyon. Right after that, TachyonFS.getFile() method is called. In all the versions of Tachyon I tried (0.4.1, 0.4.0), the second method will return a

Cannot create dir in Tachyon when running Spark with OFF_HEAP caching (FileDoesNotExistException)

2014-07-07 Thread Teng Long
Hi guys, I'm running Spark 1.0.0 with Tachyon 0.4.1, both in single node mode. Tachyon's own tests (./bin/tachyon runTests) works good, and manual file system operation like mkdir works well. But when I tried to run a very simple Spark task with RDD persist as OFF_HEAP, I got the following