collect() returns the contents of the RDD back to the Driver in a local
variable. Where is the local variable?
Try
val result = rdd.map(x => x + 1).collect()
regards,
Apostolos
On 21/2/20 21:28, Nikhil Goyal wrote:
Hi all,
I am trying to use almond scala kernel to run spark session on
Hi all,
I am trying to use almond scala kernel to run spark session on Jupyter. I
am using scala version 2.12.8. I am creating spark session with master set
to Yarn.
This is the code:
val rdd = spark.sparkContext.parallelize(Seq(1, 2, 4))
rdd.map(x => x + 1).collect()
Exception:
d Boolean, which are serializable by
default. So you can change the definition to function, instead of method, which
should work.
Yong
From: Darshan Pandya <darshanpan...@gmail.com>
Sent: Friday, February 17, 2017 10:36 PM
To: user
Subject: Serialization e
Hi Darshan ,
When you get org.apache.spark.SparkException: Task not serializable
exception, it means that you are using a reference to an instance of a
non-serialize class inside a transformation.
Hope following link will help.
Hello,
I am getting the famous serialization exception on running some code as
below,
val correctColNameUDF = udf(getNewColumnName(_: String, false:
Boolean): String);
val charReference: DataFrame = thinLong.select("char_name_id",
"char_name").withColumn("columnNameInDimTable",
Hi having some problems with the piece of code I inherited:
the error messages i get are:
the code runs if i exclude the following line:
any help appreciated.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/serialization-error-tp25131.html
Sent
>
>
> the code runs if i exclude the following line:
>
>
> any help appreciated.
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/serialization-error-tp25131.html
> Se
:
>
>
> the code runs if i exclude the following line:
>
>
> any help appreciated.
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/serialization-error-tp25131.ht
Hi, I’m receiving a task not serializable exception using Spark GraphX (Scala
2.11.6 / JDK 1.8 / Spark 1.5)
My vertex data is of type (VertexId, immutable Set),
My edge data is of type PartialFunction[ISet[E], ISet[E]] where each ED has a
precomputed function.
My vertex program:
val
I run the MovieLensALS, but meet the following error. The weird thing is
that this issue only appear under openjdk. And this is based on the 1.5, I
found several related tickets, not sure has anyone else meet the same issue
and know the solution ? Thanks
private HTable table;
You should declare table variable within apply() method.
BTW which hbase release are you using ?
I see you implement caching yourself. You can make use of the following
HTable method:
public void setWriteBufferSize(long writeBufferSize) throws
Can you show us the code for loading Hive into hbase ?
There shouldn't be 'return' statement in that code.
Cheers
On Jun 20, 2015, at 10:10 PM, Nishant Patel nishant.k.pa...@gmail.com wrote:
Hi,
I am loading data from Hive table to Hbase after doing some manipulation.
I am getting
Hi,
I am loading data from Hive table to Hbase after doing some manipulation.
I am getting error as 'Task not Serializable'.
My code is as below.
public class HiveToHbaseLoader implements Serializable {
public static void main(String[] args) throws Exception {
String
arguments are values of it. The name of the argument is important and all
you need to do is specify those when your creating SparkConf object.
Glad it worked.
On Tue, Apr 28, 2015 at 5:20 PM, madhvi madhvi.gu...@orkash.com wrote:
Thankyou Deepak.It worked.
Madhvi
On Tuesday 28 April 2015
Thankyou Deepak.It worked.
Madhvi
On Tuesday 28 April 2015 01:39 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) wrote:
val conf = new SparkConf()
.setAppName(detail)
.set(spark.serializer,
org.apache.spark.serializer.KryoSerializer)
.set(spark.kryoserializer.buffer.mb,
arguments.get(buffersize).get)
val conf = new SparkConf()
.setAppName(detail)
.set(spark.serializer, org.apache.spark.serializer.KryoSerializer)
.set(spark.kryoserializer.buffer.mb, arguments.get(buffersize
).get)
.set(spark.kryoserializer.buffer.max.mb, arguments.get(
maxbuffersize).get)
On Tuesday 28 April 2015 01:39 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) wrote:
val conf = new SparkConf()
.setAppName(detail)
.set(spark.serializer,
org.apache.spark.serializer.KryoSerializer)
.set(spark.kryoserializer.buffer.mb,
arguments.get(buffersize).get)
Hi,
While connecting to accumulo through spark by making sparkRDD I am
getting the following error:
object not serializable (class: org.apache.accumulo.core.data.Key)
This is due to the 'key' class of accumulo which does not implement
serializable interface.How it can be solved and accumulo
18 matches
Mail list logo
