Oh, just noticed that you were calling |sc.setSystemProperty|. Actually
you need to set this property in SparkConf or in spark-defaults.conf.
And there are two configurations related to Kryo buffer size,
* spark.kryoserializer.buffer.mb, which is the initial size, and
*
Yes, it works after I append the two properties in spark-defaults.conf.
As I use python programing on spark platform,the python api does not have
SparkConf api.
Thanks.
2015-03-25 21:07 GMT+08:00 Cheng Lian lian.cs@gmail.com:
Oh, just noticed that you were calling sc.setSystemProperty.
Here is the full track
15/03/25 17:48:34 WARN TaskSetManager: Lost task 0.0 in stage 1.0 (TID 1,
cloud1): com.esotericsoftware.kryo.KryoException: Buffer overflow.
Available: 0, required: 39135
at com.esotericsoftware.kryo.io.Output.require(Output.java:138)
at
Could you please provide the full stack trace?
On 3/25/15 6:26 PM, 李铖 wrote:
It is ok when I do query data from a small hdfs file.
But if the hdfs file is 152m,I got this exception.
I try this code
.'sc.setSystemProperty(spark.kryoserializer.buffer.mb,'256')'.error
still.
```
One more exception.How to fix it .Anybody help me ,please.
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output
location for shuffle 0
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$1.apply(MapOutputTracker.scala:386)
at
Would you mind running again to see if this exception can be reproduced
again, since exception in MapOutputTracker seldom occurs, maybe some other
exceptions which lead to this error.
Thanks
Jerry
2015-03-26 10:55 GMT+08:00 李铖 lidali...@gmail.com:
One more exception.How to fix it .Anybody help
