Does your RDD contain a null key?
On Sat, Nov 8, 2014 at 11:15 AM, Bahubali Jain bahub...@gmail.com wrote:
Hi,
I have a custom key class.In this class equals() and hashcode() have been
overridden.
I have a javaPairRDD which has this class as the key .When groupbykey() or
reducebykey() is
Hi,
I'm using breeze.stats.distributions.Binomial with spark 1.1.0 and having
the same error.
I tried to add the dependency to math3 with versions 3.11, 3.2, 3.3 and it
didn't help.
Any ideas what might be the problem?
Thanks,
Lev.
anny9699 wrote
I use the breeze.stats.distributions.Bernoulli
So it seems that this problem was related to
http://apache-spark-developers-list.1001551.n3.nabble.com/Lost-executor-on-YARN-ALS-iterations-td7916.html
and increasing the executor memory worked for me.
__
Hi,
I am getting
Are there debian packages for spark?
If not I plan on making one… I threw one together in about 20 minutes as
they are somewhat easy with maven and jdeb. But of course there are other
things I need to install like cassandra support and an init script.
So I figured I’d ask here first.
If not we
lev wrote
I'm using breeze.stats.distributions.Binomial with spark 1.1.0 and having
the same error.
I tried to add the dependency to math3 with versions 3.11, 3.2, 3.3 and it
didn't help.
I am experiencing likewise with all the breeze.stats.distributions using any
math3 version.
I run
Nice! Not sure how I missed that. Building it now. If it has all the
init scripts and config in the right place I might use that.
I might have to build a cassandra package too which adds cassandra
support.. I *think* at least.
Maybe distribute this .deb with the standard downloads?
Kevin
On
looks like it doesn’t work:
[ERROR] Failed to execute goal org.vafer:jdeb:0.11:jdeb (default) on
project spark-assembly_2.10: Failed to create debian package
/Users/burton/Dropbox/projects-macbook-pro-2013-09/spark-1.1.0/assembly/target/spark_1.1.0-${buildNumber}_all.deb:
Could not create deb
OK… here’s my version.
https://github.com/spinn3r/spark-deb
it’s just two files really. so if the standard spark packages get fixed
I’ll just switch to them.
Doesn’t look like there’s an init script and the conf isn’t in /etc …
On Sat, Nov 8, 2014 at 12:06 PM, Kevin Burton bur...@spinn3r.com
Hi Lev,
I also finally couldn't solve that problem and switched to Java.util.Random.
Thanks~
Anny
On Sat, Nov 8, 2014 at 4:21 AM, lev [via Apache Spark User List]
ml-node+s1001560n18406...@n3.nabble.com wrote:
Hi,
I'm using breeze.stats.distributions.Binomial with spark 1.1.0 and having
Another note for the official debs. ‘spark’ is a bad package name because
of confusion with the spark programming lang based on ada.
There are packages for this already named ‘spark’
so I put mine as ‘apache-spark’
On Sat, Nov 8, 2014 at 12:21 PM, Kevin Burton bur...@spinn3r.com wrote:
OK…
This means you haven't actually included commons-math3 in your
application. Check the contents of your final app jar and then go
check your build file again.
On Sat, Nov 8, 2014 at 12:20 PM, lev kat...@gmail.com wrote:
Hi,
I'm using breeze.stats.distributions.Binomial with spark 1.1.0 and
The building of the Debian package in Spark works just fine for me -- I
just did it using a clean check-out of 1.1.1-SNAPSHOT and `mvn -U -Pdeb
-DskipTests clean package`. There's likely something else amiss in your
build.
Actually, that's not quite true. There is one small problem with the
I am a Spark newbie and I use python (pyspark). I am trying to run a program
on a 64 core system, but no matter what I do, it always use 1 core. It
doesn't matter if I run it using spark-submit --master local[64] run.sh or
I call x.repartition(64) in my code with an RDD, the spark program always
I am a Spark newbie and I use python (pyspark). I am trying to run a
program on a 64 core system, but no matter what I do, it always uses 1
core. It doesn't matter if I run it using spark-submit --master local[64]
run.sh or I call x.repartition(64) in my code with an RDD, the spark
program always
Weird… I’m using a 1.1.0 source tar.gz …
but if it’s fixed in 1.1.1 that’s good.
On Sat, Nov 8, 2014 at 2:08 PM, Mark Hamstra m...@clearstorydata.com
wrote:
The building of the Debian package in Spark works just fine for me -- I
just did it using a clean check-out of 1.1.1-SNAPSHOT and `mvn
I have an exception when I am trying to run a simple where clause query. I
can see the name attribute is present in the schema but somehow it still
throws the exception.
query = select name from business where business_id= + business_id
what am I doing wrong ?
thx
srinivas
Exception in thread
hi,
what would be the syntax for check for an attribute in an array data type
for my where clause ?
select * from business where cateogories contains 'X' // something like
this , is this right syntax ??
attribute: categories
type: Array
thx
srinivas
I ran into this problem too and I know of a workaround but don't exactly
know what is happening. The work around is to explicitly add either the
commons math jar or your application jar (shaded with commons math)
to spark.executor.extraClassPath.
My hunch is that this is related to the class
RDD NameStorage LevelCached PartitionsFraction CachedSize in MemorySize in
TachyonSize on Disk 8
http://hadoop-s1.oculus.guest:4042/storage/rdd?id=8 Memory Deserialized
1x Replicated 426 107% 59.7 GB 0.0 B 0.0 BAnyone understand what it means
to have more than 100% of an rdd cached?
Thanks,
It might mean that some partition was computed on two nodes, because a task for
it wasn't able to be scheduled locally on the first node. Did the RDD really
have 426 partitions total? You can click on it and see where there are copies
of each one.
Matei
On Nov 8, 2014, at 10:16 PM, Nathan
20 matches
Mail list logo
