Spark checkpoint problem

I am test checkpoint to understand how it works, My code as following: scala> val data = sc.parallelize(List("a", "b", "c")) data: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[0] at parallelize at :15 scala> sc.setCheckpointDir("/tmp/checkpoint") 15/11/25 18:09:07 WARN

Spark checkpoint problem

Hi, I am test checkpoint to understand how it works, My code as following: scala> val data = sc.parallelize(List("a", "b", "c")) data: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[0] at parallelize at :15 scala> sc.setCheckpointDir("/tmp/checkpoint") 15/11/25 18:09:07 WARN

Re:RE: Spark checkpoint problem

Spark 1.5.2. 在 2015-11-26 13:19:39，"张志强(旺轩)" <zzq98...@alibaba-inc.com> 写道： What’s your spark version? 发件人: wyphao.2007 [mailto:wyphao.2...@163.com] 发送时间: 2015年11月26日 10:04 收件人: user 抄送:dev@spark.apache.org 主题: Spark checkpoint problem I am test checkpoint to understand how i

Re:Re:Driver memory leak?

-dc5aa97c2554.png At 2015-04-29 18:48:33,zhangxiongfei zhangxiongfei0...@163.com wrote: The mount of memory that the driver consumes depends on your program logic,did you try to collect the result of Spark job? At 2015-04-29 18:42:04, wyphao.2007 wyphao.2...@163.com wrote: Hi, Dear

Re:Re: java.lang.StackOverflowError when recovery from checkpoint in Streaming

On Tue, Apr 28, 2015 at 7:35 AM, wyphao.2007 wyphao.2...@163.com wrote: Hi everyone, I am using val messages = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topicsSet) to read data from kafka(1k/second), and store the data in windows,the code

java.lang.StackOverflowError when recovery from checkpoint in Streaming

Hi everyone, I am using val messages = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topicsSet) to read data from kafka(1k/second), and store the data in windows,the code snippets as follow:val windowedStreamChannel =

Question about recovery from checkpoint exception[SPARK-6892]

Hi, When I recovery from checkpoint in yarn-cluster mode using Spark Streaming, I found it will reuse the application id (In my case is application_1428664056212_0016) before falied to write spark eventLog, But now my application id is application_1428664056212_0017,then spark write

Re:Re: Question about recovery from checkpoint exception[SPARK-6892]

to application_1428664056212_0017 not application_1428664056212_0016. At 2015-04-20 11:46:12,Sean Owen so...@cloudera.com wrote: This is why spark.hadoop.validateOutputSpecs exists, really: https://spark.apache.org/docs/latest/configuration.html On Mon, Apr 20, 2015 at 3:40 AM, wyphao.2007 wyphao.2

How to get removed RDD from windows？

I want to get removed RDD from windows as follow, The old RDDs will removed from current window, // _ // | previous window _|___ // |___| current window| -- Time //

Use mvn to build Spark 1.2.0 failed

Hi all, Today download Spark source from http://spark.apache.org/downloads.html page, and I use ./make-distribution.sh --tgz -Phadoop-2.2 -Pyarn -DskipTests -Dhadoop.version=2.2.0 -Phive to build the release, but I encountered an exception as follow: [INFO] ---

Re:Re: Announcing Spark 1.2!

In the http://spark.apache.org/downloads.html page,We cann't download the newest Spark release. At 2014-12-19 17:55:29,Sean Owen so...@cloudera.com wrote: Tag 1.2.0 is older than 1.2.0-rc2. I wonder if it just didn't get updated. I assume it's going to be 1.2.0-rc2 plus a few commits

network.ConnectionManager error

Hi, When I run spark job on yarn,and the job finished success,but I found there are some error logs in the logfile as follow(the red color text): 14/09/17 18:25:03 INFO ui.SparkUI: Stopped Spark web UI at http://sparkserver2.cn:63937 14/09/17 18:25:03 INFO scheduler.DAGScheduler: Stopping

How to use jdbcRDD in JAVA

Hi,I want to know how to use jdbcRDD in JAVA not scala, trying to figure out the last parameter in the constructor of jdbcRDD thanks