RE: underlying checkpoint

2017-07-16 Thread Mendelson, Assaf
does). Thanks, Assaf. From: Bernard Jesop [mailto:bernard.je...@gmail.com] Sent: Thursday, July 13, 2017 6:58 PM To: Vadim Semenov Cc: user Subject: Re: underlying checkpoint Thank you, one of my mistakes was to think that show() was an action. 2017-07-13 17:52 GMT+02:00 Vadim

Re: underlying checkpoint

2017-07-13 Thread Bernard Jesop
Thank you, one of my mistakes was to think that show() was an action. 2017-07-13 17:52 GMT+02:00 Vadim Semenov : > You need to trigger an action on that rdd to checkpoint it. > > ``` > scala>spark.sparkContext.setCheckpointDir(".") > > scala>val df =

Re: underlying checkpoint

2017-07-13 Thread Vadim Semenov
You need to trigger an action on that rdd to checkpoint it. ``` scala>spark.sparkContext.setCheckpointDir(".") scala>val df = spark.createDataFrame(List(("Scala", 35), ("Python", 30), ("R", 15), ("Java", 20))) df: org.apache.spark.sql.DataFrame = [_1: string, _2: int] scala>

underlying checkpoint

2017-07-13 Thread Bernard Jesop
Hi everyone, I just tried this simple program : * import org.apache.spark.sql.SparkSession object CheckpointTest extends App { val spark = SparkSession .builder() .appName("Toto") .getOrCreate() spark.sparkContext.setCheckpointDir(".") val df =