date:20140811

collect failed for unknow reason when deploy use standalone mode

2014-08-11 Thread wan...@testbird.com








Hi ,? ? I use spark 0.9 run a simple computation, but it failled when I use 
standalone mode
code:? ???val sc = new SparkContext(args(0), "BayesAnalysis", 
System.getenv("SPARK_HOME"), SparkContext.jarOfClass(this.getClass).toSeq)?

? ??val dataSet = sc.textFile(args(1)).map(_.split(",")).filter(_.length == 
14).collect;

? ??for(record <- dataSet){

??? println(record.mkString(" "))

? ??}

run on local mode, it sucessfully done.run on standalone mode, it failedjava 
-classpath realclaspath mainclass??sparkMaster hdfsFile
14/08/11 15:10:07 INFO scheduler.DAGScheduler: Failed to run collect at 
BayesAnalysis.scala:416

Exception in thread "main" org.apache.spark.SparkException: Job aborted: Task 
0.0:0 failed 4 times (most recent failure: unknown)

at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1028)

at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1026)

at 
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)

at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)

at 
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$abortStage(DAGScheduler.scala:1026)

at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:619)

at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:619)

at scala.Option.foreach(Option.scala:236)

at 
org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:619)

at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$anonfun$receive$1.applyOrElse(DAGScheduler.scala:207)

at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)

at akka.actor.ActorCell.invoke(ActorCell.scala:456)

at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)

at akka.dispatch.Mailbox.run(Mailbox.scala:219)

at 
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)

at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)

at 
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)

at 
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)

at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)


公司：TestBird??
QQ：2741334465
Email: wan...@testbird.com
地址：成都市高新区天府软件园C8-3#??邮编：610041
网址：http://www.testbird.com?

Re: collect failed for unknow reason when deploy use standalone mode

2014-08-11 Thread jeanlyn92

Hi wangyi:
have more detail information？
I guess it maybe caused by need a jars that havn't upload to the
workers,such as your main class.

./bin/spark-class org.apache.spark.deploy.Client launch
   [client-options] \
  \
   [application-options]


application-jar-url: Path to a bundled jar including your application
and all dependencies. Currently, the URL must be globally visible
inside of your cluster, for instance, an `hdfs://` path or a `file://`
path that is present on all nodes.





Hi ,
I use spark 0.9 run a simple computation, but it failled when I use
standalone mode

code:
* val sc = new SparkContext(args(0), "BayesAnalysis",
System.getenv("SPARK_HOME"), SparkContext.jarOfClass(this.getClass).toSeq) *




*val dataSet = sc.textFile(args(1)).map(_.split(",")).filter(_.length
== 14).collect; for(record <- dataSet){
println(record.mkString(" ")) }*

run on local mode, it sucessfully done.
run on standalone mode, it failed
java -classpath realclaspath mainclass  sparkMaster hdfsFile


*14/08/11 15:10:07 INFO scheduler.DAGScheduler: Failed to run collect at
BayesAnalysis.scala:416 Exception in thread "main"
org.apache.spark.SparkException: Job aborted: Task 0.0:0 failed 4 times
(most recent failure: unknown) *
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1028)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1026)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.org

$apache$spark$scheduler$DAGScheduler$$abortStage(DAGScheduler.scala:1026)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:619)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:619)
at scala.Option.foreach(Option.scala:236)
at
org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:619)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$anonfun$receive$1.applyOrElse(DAGScheduler.scala:207)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)


2014-08-11 15:41 GMT+08:00 jeanlyn92 :

> Hi wangyi:
> have more detail information？
> I guess it maybe cause by need a jars that havn't upload to the works.
>
> ./bin/spark-class org.apache.spark.deploy.Client launch
>[client-options] \
>   \
>[application-options]
>
>
> application-jar-url: Path to a bundled jar including your application and all 
> dependencies. Currently, the URL must be globally visible inside of your 
> cluster, for instance, an `hdfs://` path or a `file://` path that is present 
> on all nodes.
>
>
>
>
>
> Hi ,
> I use spark 0.9 run a simple computation, but it failled when I use
> standalone mode
>
> code:
> * val sc = new SparkContext(args(0), "BayesAnalysis",
> System.getenv("SPARK_HOME"), SparkContext.jarOfClass(this.getClass).toSeq) *
>
>
>
>
> *val dataSet = sc.textFile(args(1)).map(_.split(",")).filter(_.length
> == 14).collect;  for(record <- dataSet){
> println(record.mkString(" "))  }*
>
> run on local mode, it sucessfully done.
> run on standalone mode, it failed
> java -classpath realclaspath mainclass  sparkMaster hdfsFile
>
>
> *14/08/11 15:10:07 INFO scheduler.DAGScheduler: Failed to run collect at
> BayesAnalysis.scala:416 Exception in thread "main"
> org.apache.spark.SparkException: Job aborted: Task 0.0:0 failed 4 times
> (most recent failure: unknown) *
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1028)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1026)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> at org.apache.spark.scheduler.DAGScheduler.org
> $apache$spark$scheduler$DAGScheduler$$abortStage(DAGScheduler.scala:1026)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:619)
> at
> org.apache.spark.scheduler.DA

Spark RuntimeException due to Unsupported datatype NullType

2014-08-11 Thread rafeeq s

Hi,

*Spark RuntimeException due to Unsupported datatype NullType , *When saving
null primitives *jsonRDD *with *.saveAsParquetFile()*

*Code: I am trying to* store jsonRDD into Parquet file using *saveAsParquetFile
with below code.*

JavaRDD javaRDD = ssc.sparkContext().parallelize(jsonData);
JavaSchemaRDD schemaObject = sqlContext.jsonRDD(javaRDD);
*schemaObject.saveAsParquetFile*("tweets/tweet" + time.toString().replace("
ms", "") + ".parquet");

*Input: *In below *JSON input* have some *null values* which are not
supported by spark and throwing error as *Unsupported datatype NullType.*

{"id":"tag:search.twitter.com
,2005:11","objectType":"activity","actor":{"objectType":"person","id":"id:
twitter.com:111","link":"http://www.twitter.com/funtubevids","displayName":"مشاهد
حول العالم","postedTime":"2014-05-01T06:14:51.000Z","image":"
https://pbs.twimg.com/profile_images/111/VORNn-Df_normal.png";,
*"summary"*:*null*,"links":[{*"href":null*
,"rel":"me"}],"friendsCount":0,"followersCount":49,"listedCount":0,"statusesCount":61,
*"twitterTimeZone":null*,"verified":false*,"utcOffset":null*
,"preferredUsername":"funtubevids","languages":["en"],"favoritesCount":0},"verb":"post","postedTime":"2014-05-27T17:33:54.000Z","generator":{"displayName":"web","link":"
http://twitter.com
"},"provider":{"objectType":"service","displayName":"Twitter","link":"
http://www.twitter.com"},"link":";
http://twitter.com/funtubevids/statuses/1","body":"القيادة في
مدرج الطيران #مهبط #مدرج #مطار #هبوط #قيادة #سيارة #طائرة #airport #plane
#car https://t.co/gnn7LKE6pC","object":"urls":[{"url":";
https://t.co/gnn7LKE6pC","expanded_url":";
https://www.youtube.com/watch?v=J-j6RSRMvRo
","expanded_status":200}],"klout_score":10,"language":{"value":"ar"}}}


*ERROR* scheduler.JobScheduler: Error running job streaming job
140774119 ms.0
*java.lang.RuntimeException: Unsupported datatype NullType*
   at scala.sys.package$.error(package.scala:27)
   at
org.apache.spark.sql.parquet.ParquetTypesConverter$.fromDataType(ParquetTypes.scala:267)
   at
org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$2.apply(ParquetTypes.scala:244)
   at
org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$2.apply(ParquetTypes.scala:244)
   at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
   at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
   at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
   at
scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
   at
scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
   at
scala.collection.AbstractTraversable.map(Traversable.scala:105)
   at
org.apache.spark.sql.parquet.ParquetTypesConverter$.fromDataType(ParquetTypes.scala:243)
   at
org.apache.spark.sql.parquet.ParquetTypesConverter$.fromDataType(ParquetTypes.scala:235)
   at
org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$2.apply(ParquetTypes.scala:244)
   at
org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$2.apply(ParquetTypes.scala:244)
   at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
   at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
   at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
   at
scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
   at
scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
   at
scala.collection.AbstractTraversable.map(Traversable.scala:105)
   at
org.apache.spark.sql.parquet.ParquetTypesConverter$.fromDataType(ParquetTypes.scala:243)
   at
org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$3.apply(ParquetTypes.scala:287)
   at
org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$3.apply(ParquetTypes.scala:286)
   at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
   at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
   at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
   at
scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
   at
scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
   at
scala.collection.AbstractTraversable.map(Traversable.scala:105)
   at
org.apache.spark.sql.parquet.ParquetTypesConverter$.convertFromAttributes(ParquetTypes.scala:285)
   at
org.apache.spark.sql.parquet.ParquetTypesConverter$.writeMetaData(ParquetTypes.scala:331)
   at
org.apache.spark.sql.parquet.

Re: error with pyspark

2014-08-11 Thread Ron Gonzalez

If you're running on Ubuntu, do ulimit -n, which gives the max number of 
allowed open files. You will have to change the value in 
/etc/security/limits.conf to something like 1, logout and log back in.

Thanks,
Ron

Sent from my iPad

> On Aug 10, 2014, at 10:19 PM, Davies Liu  wrote:
> 
>> On Fri, Aug 8, 2014 at 9:12 AM, Baoqiang Cao  wrote:
>> Hi There
>> 
>> I ran into a problem and can’t find a solution.
>> 
>> I was running bin/pyspark < ../python/wordcount.py
> 
> you could use bin/spark-submit  ../python/wordcount.py
> 
>> The wordcount.py is here:
>> 
>> 
>> import sys
>> from operator import add
>> 
>> from pyspark import SparkContext
>> 
>> datafile = '/mnt/data/m1.txt'
>> 
>> sc = SparkContext()
>> outfile = datafile + '.freq'
>> lines = sc.textFile(datafile, 1)
>> counts = lines.flatMap(lambda x: x.split(' ')) \
>>.map(lambda x: (x, 1)) \
>>.reduceByKey(add)
>> output = counts.collect()
>> 
>> outf = open(outfile, 'w')
>> 
>> for (word, count) in output:
>>   outf.write(word.encode('utf-8') + '\t' + str(count) + '\n')
>> outf.close()
>> 
>> 
>> 
>> The error message is here:
>> 
>> 14/08/08 16:01:59 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 0)
>> java.io.FileNotFoundException:
>> /tmp/spark-local-20140808160150-d36b/12/shuffle_0_0_468 (Too many open
>> files)
> 
> This message means that the Spark (JVM) had reach  the max number of open 
> files,
> there are fd leak some where, unfortunately I can not reproduce this
> problem.  What
> is the version of Spark?
> 
>>at java.io.FileOutputStream.open(Native Method)
>>at java.io.FileOutputStream.(FileOutputStream.java:221)
>>at
>> org.apache.spark.storage.DiskBlockObjectWriter.open(BlockObjectWriter.scala:107)
>>at
>> org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:175)
>>at
>> org.apache.spark.shuffle.hash.HashShuffleWriter$$anonfun$write$1.apply(HashShuffleWriter.scala:67)
>>at
>> org.apache.spark.shuffle.hash.HashShuffleWriter$$anonfun$write$1.apply(HashShuffleWriter.scala:65)
>>at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>>at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>>at
>> org.apache.spark.shuffle.hash.HashShuffleWriter.write(HashShuffleWriter.scala:65)
>>at
>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
>>at
>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
>>at org.apache.spark.scheduler.Task.run(Task.scala:54)
>>at
>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:199)
>>at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>at java.lang.Thread.run(Thread.java:744)
>> 
>> 
>> The m1.txt is about 4G, and I have >120GB Ram and used -Xmx120GB. It is on
>> Ubuntu. Any help please?
>> 
>> Best
>> Baoqiang Cao
>> Blog: http://baoqiang.org
>> Email: bqcaom...@gmail.com
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
> 

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

spark sql (can it call impala udf)

2014-08-11 Thread marspoc

I want to do the below query that I run in impala calling a c++ UDF in spark
sql.
In which pnl_flat_pp and pfh_flat are both impala table with partitioned.

Can Spark Sql does that?



select a.pnl_type_code,percentile_udf_cloudera(cast(90.0 as
double),sum(pnl_vector1),sum(pnl_vector2),sum(pnl_vector3),sum(pnl_vector4),sum(pnl_vector5),sum(pnl_vector6),sum(pnl_vector7),sum(pnl_vector8),sum(pnl_vector9),sum(pnl_vector10),sum(pnl_vector11),sum(pnl_vector12),sum(pnl_vector13),sum(pnl_vector14))
FROM ibrisk.pnl_flat_pp a JOIN(select portfolio_code from ibrisk.pfh_flat
where pl0_code = '"3"') b ON a.portfolio_code = b.portfolio_code where
rf_level = '"0"' and calc_ref = "7020704" and excl_pnl != '"1"' group by
a.pnl_type_code



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/spark-sql-can-it-call-impala-udf-tp11878.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

[spark-streaming] kafka source and flow control

2014-08-11 Thread gpasquiers

Hi,

I’m new to this mailing list as well as spark-streaming.

I’m using spark-streaming in a cloudera environment to consume a kafka
source and store all data into hdfs. There is a great volume of data and our
issue is that the kafka consumer is going too fast for HDFS, it fills up the
storage (memory) of spark-streaming, and makes it drop messages that have
not been saved to HDFS yet…

I’m not at all in an optimal environment regarding performances (spark
running in a VM, accessing HDFS through an SSH tunnel), but I think it’s not
a bad setup to test my system’s reliability.

I’ve been (unsuccessfully) looking for a way to slow down my kafka consumer
depending on the system’s health, maybe make the store() operation block if
there is no available room for storage.

I have complete control over the kafka consumer since I developed a custom
Receiver as a workaround to :
https://issues.apache.org/jira/browse/SPARK-2103 but i’d like to make a flow
control more intelligent than a simple rate limite (x messages or bytes per
second).

I’m interested in all ideas or suggestions.

B.R.




--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-kafka-source-and-flow-control-tp11879.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: CDH5, HiveContext, Parquet

2014-08-11 Thread chutium

hive-thriftserver does not work with parquet tables in hive metastore also,
this PR will fix it too?

do not need to change any pom.xml ?



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/CDH5-HiveContext-Parquet-tp11853p11880.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Low Performance of Shark over Spark.

2014-08-11 Thread vinay . kashyap

Hi Yana,
I notice there is GC happening in every executor which
is around 400ms on an average. Do you think it is a major impact on the
overall query time..??
And regarding the memory for a single worker,
I have tried distributing the memory by increasing the number of workers
per node and dividing the memory for each worker accordingly.
Means,
in my trials I have configured 96G, 48G for each worker, but dint see any
difference in the query time.
You have any comments regarding the GC
time..?? Because, as I have mentioned in the earlier mails, I have varied
the memory fractions but not much difference was
seen.
Thanks and regards
Vinay Kashyap

From:"Yana Kadiyska"

Sent:"vinay.kashyap"

Date:Sat, August 9, 2014 6:56 am

Subject:Re: Low Performance of Shark over Spark.

Can you see where your time is spent? I have noticed quite a bit of
variance in query time in the case where GC occurs in the middle of a
computation. I'm guessing you're talking time averaged over multiple runs
but thought I'd mention this as a possible thing to check. I might be
mistaken but 96G in a single worker(if I'm reading your mail correctly)
seems on the high side (although I cannot find now any recommendation on
what it should be)

76 matches

Mail list logo