spark.speculation was not set, any speculative execution on tachyon side? tachyon-env.sh only changed following
export TACHYON_MASTER_ADDRESS=test01.zala #export TACHYON_UNDERFS_ADDRESS=$TACHYON_HOME/underfs export TACHYON_UNDERFS_ADDRESS=hdfs://test01.zala:8020 export TACHYON_WORKER_MEMORY_SIZE=16GB test01.zala is master node for HDFS, tachyon, Spark, etc. worker nodes are test02.zala test03.zala test04.zala spark-shell run on test02 after parquetFile.saveAsParquetFile("tachyon://test01.zala:19998/parquet_1") i got FailedToCheckpointException with Failed to rename but tfs lsr, there are some temporary files and metadata file 0.00 B 08-11-2014 16:19:28:054 /parquet_1 881.00 B 08-11-2014 16:19:28:054 In Memory /parquet_1/_metadata 0.00 B 08-11-2014 16:19:28:314 /parquet_1/_temporary 0.00 B 08-11-2014 16:19:28:314 /parquet_1/_temporary/0 0.00 B 08-11-2014 16:19:28:931 /parquet_1/_temporary/0/_temporary 0.00 B 08-11-2014 16:19:28:931 /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000000_33 0.00 B 08-11-2014 16:19:28:931 In Memory /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000000_33/part-r-1.parquet 0.00 B 08-11-2014 16:19:28:940 /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000001_35 0.00 B 08-11-2014 16:19:28:940 In Memory /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000001_35/part-r-2.parquet 0.00 B 08-11-2014 16:19:28:962 /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000003_36 0.00 B 08-11-2014 16:19:28:962 In Memory /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000003_36/part-r-4.parquet 0.00 B 08-11-2014 16:19:28:971 /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000002_34 0.00 B 08-11-2014 16:19:28:971 In Memory /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000002_34/part-r-3.parquet 0.00 B 08-11-2014 16:20:06:349 /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000001_37 0.00 B 08-11-2014 16:20:06:349 In Memory /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000001_37/part-r-2.parquet 0.00 B 08-11-2014 16:20:09:519 /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000003_38 0.00 B 08-11-2014 16:20:09:519 In Memory /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000003_38/part-r-4.parquet 0.00 B 08-11-2014 16:20:18:777 /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000002_39 0.00 B 08-11-2014 16:20:18:777 In Memory /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000002_39/part-r-3.parquet 0.00 B 08-11-2014 16:20:28:315 /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000000_40 0.00 B 08-11-2014 16:20:28:315 In Memory /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000000_40/part-r-1.parquet 0.00 B 08-11-2014 16:20:38:382 /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000001_41 0.00 B 08-11-2014 16:20:38:382 In Memory /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000001_41/part-r-2.parquet 0.00 B 08-11-2014 16:20:40:681 /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000003_42 0.00 B 08-11-2014 16:20:40:681 In Memory /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000003_42/part-r-4.parquet 0.00 B 08-11-2014 16:20:50:376 /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000002_43 0.00 B 08-11-2014 16:20:50:376 In Memory /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000002_43/part-r-3.parquet 0.00 B 08-11-2014 16:21:00:932 /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000000_44 0.00 B 08-11-2014 16:21:00:932 In Memory /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000000_44/part-r-1.parquet 0.00 B 08-11-2014 16:21:10:355 /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000001_45 0.00 B 08-11-2014 16:21:10:355 In Memory /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000001_45/part-r-2.parquet 0.00 B 08-11-2014 16:21:11:468 /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000003_46 0.00 B 08-11-2014 16:21:11:468 In Memory /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000003_46/part-r-4.parquet 0.00 B 08-11-2014 16:21:21:681 /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000002_47 0.00 B 08-11-2014 16:21:21:681 In Memory /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000002_47/part-r-3.parquet 0.00 B 08-11-2014 16:21:32:583 /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000000_48 0.00 B 08-11-2014 16:21:32:583 In Memory /parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000000_48/part-r-1.parquet -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/share-reuse-off-heap-persisted-tachyon-RDD-in-SparkContext-or-saveAsParquetFile-on-tachyon-in-SQLCont-tp11897p11977.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org