you ran out of kryo buffer. are you using spark 1.1 (which supports buffer
resizing) or spark 1.0 (which has a fixed size buffer)?
On Oct 21, 2014 5:30 PM, "nitinkak001" <nitinkak...@gmail.com> wrote:
> I am running a simple rdd filter command. What does it mean?
> Here is the full stack trace(and code below it):
>
> com.esotericsoftware.kryo.KryoException: Buffer overflow. Available: 0,
> required: 133
> at com.esotericsoftware.kryo.io.Output.require(Output.java:138)
> at
> com.esotericsoftware.kryo.io.Output.writeString_slow(Output.java:420)
> at com.esotericsoftware.kryo.io.Output.writeString(Output.java:326)
> at
>
> com.esotericsoftware.kryo.serializers.DefaultArraySerializers$StringArraySerializer.write(DefaultArraySerializers.java:274)
> at
>
> com.esotericsoftware.kryo.serializers.DefaultArraySerializers$StringArraySerializer.write(DefaultArraySerializers.java:262)
> at
> com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
> at
>
> org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:138)
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:197)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
>
> *Here is the code of the main function:*
>
> /String comparisonFieldIndexes = "16,18";
> String segmentFieldIndexes = "14,15";
> String comparisonFieldWeights = "50, 50";
> String delimiter = ""+'\001';
>
> PartitionDataOnColumn parOnCol = new PartitionDataOnColumn(70,
> comparisonFieldIndexes, comparisonFieldWeights, segmentFieldIndexes,
> delimiter);
>
> JavaRDD<String> filtered_rdd = origRDD.filter(parOnCol.new
> FilterEmptyFields(parOnCol.fieldIndexes, parOnCol.DELIMITER) );
>
> parOnCol.printRDD(filtered_rdd);/
>
>
> *Here is the FilterEmptyFields class:*
>
> /public class FilterEmptyFields implements Function<String,
> Boolean>
> {
>
> final int[] nonEmptyFields;
> final String DELIMITER;
>
> public FilterEmptyFields(int[] nonEmptyFields, String
> delimiter){
> this.nonEmptyFields = nonEmptyFields;
> this.DELIMITER = delimiter;
> }
>
> @Override
> public Boolean call(String s){
>
> String[] fields = s.split(DELIMITER);
>
> for(int i=0; i<nonEmptyFields.length; i++){
> if(fields[nonEmptyFields[i]] == null ||
> fields[nonEmptyFields[i]].isEmpty()){
> return false;
> }
> }
>
> return true;
> }
>
> }</i>
>
> Any suggestions guys?
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/com-esotericsoftware-kryo-KryoException-Buffer-overflow-tp16947.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
