NoSuchElementException

My code throws an exception when I am trying to create new DataSet from within SteamWriter sink Simplified version of the code val df = sparkSession.readStream .format("json") .option("nullValue", " ") .option("headerFlag", "true") .option("spark.sql.shuffle.partitions", 1)

Re: Possible memory leak after closing spark context in v2.0.1

, Sean Owen <so...@cloudera.com> wrote: > Did you unpersist the broadcast objects? > > On Mon, Oct 17, 2016 at 10:02 AM lev <kat...@gmail.com> wrote: > >> Hello, >> >> I'm in the process of migrating my application to spark 2.0.1, >> And I think there

Possible memory leak after closing spark context in v2.0.1

Hello, I'm in the process of migrating my application to spark 2.0.1, And I think there is some memory leaks related to Broadcast joins. the application has many unit tests, and each individual test suite passes, but when running all together, it fails on OOM errors. In the begging of each

spark 2.0.0 - how to build an uber-jar?

hi, in spark 1.5, to build an uber-jar, I would just compile the code with: mvn ... package and that will create one big jar with all the dependencies. when trying to do the same with spark 2.0, I'm getting a tar.gz file instead. this is the full command I'm using: mvn -Pyarn -Phive

log4j custom appender ClassNotFoundException with spark 1.5.2

Hi, I'm using spark 1.5.2 and running on a yarn cluster and trying to use a custom log4j appender in my setup there are 3 jars: the uber jar: spark.yarn.jar=uber-jar.jar the jar that contains the main class: main.jar additional jar with dependencies: dep.jar (passed with the --jars flag to

Prevent spark from serializing some objects

serializer on production code Thanks, Lev. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Prevent-spark-from-serializing-some-objects-tp24700.html Sent from the Apache Spark User List mailing list archive at

Using native blas with mllib

netlib-native_system-linux-x86_64.so netlib-native_system-linux-x86_64.so.asc Anything else I can try? Thanks, Lev. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Using-native-blas-with-mllib-tp21156.html Sent from the Apache Spark User List

Re: Consistent hashing of RDD row

After checking the spark code, I now realize that an rdd that was cached to disk can't be evicted, so I will just persist the rdd to disk after the random numbers are created. -- View this message in context:

Consistent hashing of RDD row

evicted and recalculated, will the index stay the same? I'd love to hear your thoughts on the matter. Thanks, Lev. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Consistent-hashing-of-RDD-row-tp20820.html Sent from the Apache Spark User List mailing list

repartition combined with zipWithIndex get stuck

storage.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 1096.0 B, free 132.5 MB) Am I doing something wrong here or is it a bug? Is there some work around? Thanks, Lev. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/repartition

Re: org/apache/commons/math3/random/RandomGenerator issue

in the compiled jar, but the error is still there. it's wired that the class is not found even when it's in the jar. - adding the math3 jar to a dir that in oozie.libpath. I'm running the spark jar with oozie, but that also didn't solve it. Thanks, Lev. -- View this message in context: http

Re: org/apache/commons/math3/random/RandomGenerator issue

hdfs:// ) and it didn't work.. Thanks, Lev. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/org-apache-commons-math3-random-RandomGenerator-issue-tp15748p18453.html Sent from the Apache Spark User List mailing list archive at Nabble.com

Re: org/apache/commons/math3/random/RandomGenerator issue

Hi, I'm using breeze.stats.distributions.Binomial with spark 1.1.0 and having the same error. I tried to add the dependency to math3 with versions 3.11, 3.2, 3.3 and it didn't help. Any ideas what might be the problem? Thanks, Lev. anny9699 wrote I use the breeze.stats.distributions.Bernoulli