What is the difference between ml.classification.LogisticRegression and mllib.classification.LogisticRegressionWithLBFGS

Hi everyone, I'm curious about the difference between ml.classification.LogisticRegression and mllib.classification.LogisticRegressionWithLBFGS. Both of them are optimized using LBFGS, the only difference I see is LogisticRegression takes DataFrame while LogisticRegressionWithLBFGS takes RDD. So

RE: SparkR Error in sparkR.init(master=“local”) in RStudio

Not sure "/C/DevTools/spark-1.5.1/bin/spark-submit.cmd" is a valid? From: Hossein [mailto:fal...@gmail.com] Sent: Wednesday, October 7, 2015 12:46 AM To: Khandeshi, Ami Cc: Sun, Rui; akhandeshi; user@spark.apache.org Subject: Re: SparkR Error in sparkR.init(master=“local”) in RStudio Have you bui

Re: Help needed to reproduce bug

Hi Nick, I will try to reproduce your issue on a couple of environment. Just wanted which kind of environment you use: spark standalone, spark on yarn, or spark on mesos ? For you, does it occur with any transform() on any RDD or do you use specific RDD ? I plan to use your code in a main

Re: SparkR Error in sparkR.init(master=“local”) in RStudio

Is it possible that your user does not have permission to write temp file? On Tue, Oct 6, 2015 at 10:26 AM -0700, "akhandeshi" wrote: It seems it is failing at path <- tempfile(pattern = "backend_port") I do not see backend_port directory created... -- View this message in context: ht

Re: Notification on Spark Streaming job failure

Hi Vikram, So you give up using yarn-cluster mode of launching Spark jobs, is that right? AFAIK when using yarn-cluster mode, the launch process (spark-submit) monitors job running on YARN, but if it is killed/dies, it just stops printing the state (RUNNING usually), without influencing the monitor

Help needed to reproduce bug

Hi spark community, I was hoping someone could help me by running a code snippet below in the spark shell, and seeing if they see the same buggy behavior I see. Full details of the bug can be found in this JIRA issue I filed: https://issues.apache.org/jira/browse/SPARK-10942. The issue was closed

Re: GenericMutableRow and Row mismatch on Spark 1.5?

An approach can be to wrap your MutableRow in WrappedInternalRow which is a child class of Row. Hemant www.snappydata.io linkedin.com/company/snappydata On Tue, Oct 6, 2015 at 3:21 PM, Ophir Cohen wrote: > Hi Guys, > I'm upgrading to Spark 1.5. > > In our previous version (Spark 1.3 but it was

Re: Spark job workflow engine recommendations

Does Azkaban support scheduling long running jobs like spark steaming jobs? Will Azkaban kill a job if it's running for a long time. On Friday, August 7, 2015, Vikram Kone wrote: > Hien, > Is Azkaban being phased out at linkedin as rumored? If so, what's linkedin > going to use for workflow sche

Re: Notification on Spark Streaming job failure

We are using Monit to kick off spark streaming jobs n seems to work fine. On Monday, September 28, 2015, Chen Song wrote: > I am also interested specifically in monitoring and alerting on Spark > streaming jobs. It will be helpful to get some general guidelines or advice > on this, from people w

Deep learning example using spark

Hello Sir/Madam, I am working on the deep learning using spark. I have implemented some algorithms using spark. bot now i want to used the imageNet database in spark-1.0.4.. Can you give me some guideline or reference so i can handle the imageNet database. Thanking You, Sagar Jadhav.

Re: How to avoid Spark shuffle spill memory?

Hi unk1102, Try adding more memory to your nodes. Are you running Spark in the cloud? If so, increase the memory on your servers. Do you have default parallelism set (spark.default.parallelism)? If so, unset it, and let Spark decided how many partitions to allocate. You can also try refactoring

Re: Does feature parity exist between Spark and PySpark

Python APIs are sometimes a little behind Scala APIs. Another issue that arises sometimes is when you have dependencies on Java or Scala classes for serializing and deserializing data. Working with non-trivial Avro schemas has been a bit of a pain for me in Python due to the difficulty in dealing

Re: Does feature parity exist between Spark and PySpark

If you are using Dataframes in PySpark, then the performance will be the same as Scala. However, if you need to implement your own UDF, or run a map() against a DataFrame in Python, then you will pay the penalty for performance when executing those functions since all of your data has to go throug

Re: Does feature parity exist between Scala and Python on Spark

While I have a preference for Scala ( not surprising as a Typesafe person), the DataFrame API gives feature and performance parity for Python. The RDD API gives feature parity. So, use what makes you most successful for other reasons ;) Sent from my rotary phone. > On Oct 6, 2015, at 4:14 P

Re: Does feature parity exist between Spark and PySpark

That should have read "a lot of neat tricks", not "a lot of nest tricks". That's what I get for sending emails on my phone On Oct 6, 2015 8:32 PM, "Richard Eggert" wrote: > Since the Python API is built on top of the Scala implementation, its > performance can be at best roughly the same as

Re: Does feature parity exist between Spark and PySpark

Since the Python API is built on top of the Scala implementation, its performance can be at best roughly the same as that of the Scala API (as in the case of DataFrames and SQL) and at worst several orders of magnitude slower. Likewise, since the a Scala implementation of new features necessaril

Re: [Spark on YARN] Multiple Auxiliary Shuffle Service Versions

Thank you all for your help. *Alex Rovner* *Director, Data Engineering * *o:* 646.759.0052 * * On Tue, Oct 6, 2015 at 11:17 AM, Steve Loughran wrote: > > On 6 Oct 2015, at 01:23, Andrew Or wrote: > > Both the history server and the shuffle service are backward compat

Re: laziness in textFile reading from HDFS?

LZO files are not splittable by default but there are projects with Input and Output formats to make splittable LZO files. Check out twitter's elephantbird on GitHub El miércoles, 7 de octubre de 2015, Mohammed Guller escribió: > It is not uncommon to process datasets larger than available memor

Re: laziness in textFile reading from HDFS?

LZO files are not splittable by default but there are projects with Input and Output formats to make splittable LZO files. Check out twitter's elephantbird on GitHub El miércoles, 7 de octubre de 2015, Mohammed Guller escribió: > It is not uncommon to process datasets larger than available memor

RE: laziness in textFile reading from HDFS?

It is not uncommon to process datasets larger than available memory with Spark. I don't remember whether LZO files are splittable. Perhaps, in your case Spark is running into issues while decompressing a large LZO file. See if this helps: http://stackoverflow.com/questions/25248170/spark-hadoop

Re: does KafkaCluster can be public ?

Sure no prob. On Tue, Oct 6, 2015 at 6:35 PM, Tathagata Das wrote: > Given the interest, I am also inclining towards making it a public > developer API. Maybe even experimental. Cody, mind submitting a patch? > > > On Tue, Oct 6, 2015 at 7:45 AM, Sean Owen wrote: > >> For what it's worth, I als

Re: Does feature parity exist between Spark and PySpark

Hi 2 cents 1. It should not be true anymore if data frames are used. The reason is regardless of the language DF uses same optimization engine behind the scene. 2. This is generally true in the sense Python APIs are typically little behind of scala/java ones. Best Ayan On Wed, Oct 7, 2015 at 9

Re: does KafkaCluster can be public ?

Given the interest, I am also inclining towards making it a public developer API. Maybe even experimental. Cody, mind submitting a patch? On Tue, Oct 6, 2015 at 7:45 AM, Sean Owen wrote: > For what it's worth, I also use this class in an app, but it happens > to be from Java code where it acts

Re: Weird performance pattern of Spark Streaming (1.4.1) + direct Kafka

, Cody Koeninger >>>> wrote: >>>> >>>>> Can you say anything more about what the job is doing? >>>>> >>>>> First thing I'd do is try to get some metrics on the time taken by >>>>> your code on the executors (

Re: RDD of ImmutableList

Nobody is saying not to use immutable data structures, only that guava's aren't natively supported. Scala's default collections library is all immutable. list, Vector, Map. This is what people generally use, especially in scala code! El martes, 6 de octubre de 2015, Jakub Dubovsky < spark.dubovsk

Re: Weird performance pattern of Spark Streaming (1.4.1) + direct Kafka

ay anything more about what the job is doing? >>>> >>>> First thing I'd do is try to get some metrics on the time taken by your >>>> code on the executors (e.g. when processing the iterator) to see if it's >>>> consistent between the two

Re: laziness in textFile reading from HDFS?

Agreed. This is spark 1.2 on CDH5.x. How do you mitigate where the data sets are larger than available memory? My jobs stall and gc/heap issues all over the place. ..via mobile > On Oct 6, 2015, at 4:44 PM, Mohammed Guller wrote: > > I have not used LZO compressed files from Spark, so not s

Re: Weird performance pattern of Spark Streaming (1.4.1) + direct Kafka

Maas >>> wrote: >>> >>>> Hi, >>>> >>>> We recently migrated our streaming jobs to the direct kafka receiver. >>>> Our initial migration went quite fine but now we are seeing a weird zig-zag >>>> performance pattern we

RE: laziness in textFile reading from HDFS?

I have not used LZO compressed files from Spark, so not sure why it stalls without caching. In general, if you are going to make just one pass over the data, there is not much benefit in caching it. The data gets read anyway only after the first action is called. If you are calling just a map

Re: laziness in textFile reading from HDFS?

One. I read in LZO compressed files from HDFS Perform a map operation cache the results of this map operation call saveAsHadoopFile to write LZO back to HDFS. Without the cache, the job will stall. mn > On Oct 5, 2015, at 7:25 PM, Mohammed Guller wrote: > > Is there any specific reason for

Does feature parity exist between Spark and PySpark

Hi I'm hearing a common theme running that I should only do serious programming in Scala on Spark (1.5.1). Real power users use Scala. It is said that Python is great for analytics but in the end the code should be written to Scala to finalise. There are a number of reasons I'm hearing: 1. Spark

Does feature parity exist between Scala and Python on Spark

Hi, I'm hearing a common theme running that I should only do serious programming in Scala on Spark (1.5.1). Real power users use Scala. It is said that Python is great for analytics but in the end the code should be written to Scala to finalise. There are a number of reasons I'm hearing: 1. Spark

Help with big data operation performance

Hi all, In a stand-alone cluster operation, with more than 80 gbs of ram in each node, am trying to: 1. load a partitioned json dataframe which weights around 100GB as input 2. apply transformations such as cast some column types 3. get some percentiles which involves sort by, rdd

Re: 1.5 Build Errors

Hi All, Sean patiently worked with me in solving this issue. The problem was entirely my fault in settings MAVEN_OPTS env variable was set and was overriding everything. --Ben On Tue, Sep 8, 2015 at 1:37 PM, Benjamin Zaitlen wrote: > Yes, just reran with the following > > (spark_build)root@ip

Re: compatibility issue with Jersey2

On Tue, Oct 6, 2015 at 12:04 PM, Gary Ogden wrote: > But we run unit tests differently in our build environment, which is > throwing the error. It's setup like this: > > I suspect this is what you were referring to when you said I have a problem? Yes, that is what I was referring to. But, in your

How to avoid Spark shuffle spill memory?

Hi I have a Spark job which runs for around 4 hours and it shared SparkContext and runs many child jobs. When I see each job in UI I see shuffle spill of around 30 to 40 GB and because of that many times executors gets lost because of using physical memory beyond limits how do I avoid shuffle spill

Re: Weird performance pattern of Spark Streaming (1.4.1) + direct Kafka

;>> performance pattern we cannot explain. >>> In alternating fashion, one task takes about 1 second to finish and the >>> next takes 7sec for a stable streaming rate. >>> >>> Here are comparable metrics for two successive tasks: >>> *Slow*: >>> >

Spark cache memory storage

Hi, there My understanding is that the cache storage is calculated as following executor heap size * spark.storage.safetyFraction * spark.storage.memoryFraction. The default value for safetyFraction is 0.9 and memoryFraction is 0.6. When I started a spark job on YARN, I set executor-memory to be

Re: Weird performance pattern of Spark Streaming (1.4.1) + direct Kafka

r streaming jobs to the direct kafka receiver. >>> Our initial migration went quite fine but now we are seeing a weird zig-zag >>> performance pattern we cannot explain. >>> In alternating fashion, one task takes about 1 second to finish and the >>> next takes

Re: compatibility issue with Jersey2

In our separate environments we run it with spark-submit, so I can give that a try. But we run unit tests differently in our build environment, which is throwing the error. It's setup like this: helper = new CassandraHelper(settings.getCassandra().get()); SparkConf sparkConf = get

Re: Weird performance pattern of Spark Streaming (1.4.1) + direct Kafka

t; performance pattern we cannot explain. >> In alternating fashion, one task takes about 1 second to finish and the >> next takes 7sec for a stable streaming rate. >> >> Here are comparable metrics for two successive tasks: >> *Slow*: >> >> >> ​

Re: ORC files created by Spark job can't be accessed using hive table

Thanks Michael so the following code written using Spark 1.5.1 should be able to recognise by Hive table right dataFrame.write().mode(SaveMode.Append).partitionBy(" entity","date").format("orc").save("baseTable"); Hive console: Create external table bla bla stored as ORC Location '/user/xyz/baseT

Re: compatibility issue with Jersey2

On Tue, Oct 6, 2015 at 5:57 AM, oggie wrote: > We have a Java app written with spark 1.3.1. That app also uses Jersey 2.9 > client to make external calls. We see spark 1.4.1 uses Jersey 1.9. How is this app deployed? If it's run via spark-submit, you could use "spark.{driver,executor}.userClassP

Re: spark-submit --packages using different resolver

This is the ticket SPARK-10951 Cheers~ On Tue, Oct 6, 2015 at 11:33 AM, Jerry Lam wrote: > Hi Burak, > > Thank you for the tip. > Unfortunately it does not work. It throws: > > java.net.MalformedURLException: unknown protocol: s3n] > at > org.

Re: Broadcast var is null

This advice solved the problem: "Stop having your object extend App, and instead give it a main method." https://issues.apache.org/jira/browse/SPARK-4170 -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Broadcast-var-is-null-tp24927p24959.html Sent from th

Re: How can I disable logging when running local[*]?

Here’s an example. I echoed JAVA_OPTS so that you can see what I’ve got. Then I call ‘activator run’ in the project directory. jjones-mac:analyzer-perf jjones$ echo $JAVA_OPTS -Xmx4g -Xmx4g -Dlog4j.configuration=file:/Users/jjones/src/adaptive/adaptiveobjects/analyzer-perf/conf/log4j.propertie

Re: How can I disable logging when running local[*]?

The easiest way to control logging in spark shell is to run Logger.setLevel commands at the beginning of your program e.g. org.apache.log4j.Logger.getLogger("com.amazon").setLevel(org.apache.log4j.Level.WARN) org.apache.log4j.Logger.getLogger("com.amazonaws").setLevel(org.apache.log4j.Level.WARN)

Re: Weird performance pattern of Spark Streaming (1.4.1) + direct Kafka

t; next takes 7sec for a stable streaming rate. > > Here are comparable metrics for two successive tasks: > *Slow*: > > > ​ > > Executor IDAddressTask TimeTotal TasksFailed TasksSucceeded Tasks > 20151006-044141-2408867082-5050-21047-S0dnode-3.hdfs.private:3686

Re: How can I disable logging when running local[*]?

Try JAVA_OPTS='-Dlog4j.configuration=file:/' Internally, this is just spark.driver.extraJavaOptions, which you should be able to set in conf/spark-defaults.conf Can you provide more details how you invoke the driver? On Tue, Oct 6, 2015 at 9:48 AM, Jeff Jones wrote: > Thanks. Any chance you k

Re: SparkR Error in sparkR.init(master=“local”) in RStudio

It seems it is failing at path <- tempfile(pattern = "backend_port") I do not see backend_port directory created... -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkR-Error-in-sparkR-init-master-local-in-RStudio-tp23768p24958.html Sent from the Apache

Re: ORC files created by Spark job can't be accessed using hive table

I believe this is fixed in Spark 1.5.1 as long as the table is only using types that hive understands and is not partitioned. The problem with partitioned tables it that hive does not support dynamic discovery unless you manually run the repair command. On Tue, Oct 6, 2015 at 9:33 AM, Umesh Kacha

Re: API to run spark Jobs

Yeah I was going to suggest looking at the code too. It's a shame there isn't a page in the docs that covers the port 6066 rest api. On Tue, Oct 6, 2015 at 10:16 AM, Ted Yu wrote: > Please take a look at: > org.apache.spark.deploy.rest.RestSubmissionClient > > which is used > by core/src/main/

Re: API to run spark Jobs

Please take a look at: org.apache.spark.deploy.rest.RestSubmissionClient which is used by core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala FYI On Tue, Oct 6, 2015 at 10:08 AM, shahid qadri wrote: > hi Jeff > Thanks > More specifically i need the Rest api to submit pyspark job, can

Re: Spark 1.3.1 on Yarn not using all given capacity

3 cores* not 8 César. > Le 6 oct. 2015 à 19:08, Cesar Berezowski a écrit : > > I deployed hdp 2.3.1 and got spark 1.3.1, spark 1.4 is supposed to be > available as technical preview I think > > vendor’s forum ? you mean hortonworks' ? > > -- > Update on my info: > > Set Yarn to use 16 c

Re: API to run spark Jobs

hi Jeff Thanks More specifically i need the Rest api to submit pyspark job, can you point me to Spark submit REST api > On Oct 6, 2015, at 10:25 PM, Jeff Nadler wrote: > > > Spark standalone doesn't come with a UI for submitting jobs. Some Hadoop > distros might, for example EMR in AWS ha

Re: Spark 1.3.1 on Yarn not using all given capacity

Considering posting the question on vendor's forum. HDP 2.3 comes with Spark 1.4 if I remember correctly. On Tue, Oct 6, 2015 at 9:05 AM, czoo wrote: > Hi, > > This post might be a duplicate with updates from another one (by me), sorry > in advance > > I have an HDP 2.3 cluster running Spark 1.

Re: API to run spark Jobs

Spark standalone doesn't come with a UI for submitting jobs. Some Hadoop distros might, for example EMR in AWS has a job submit UI. Spark submit just calls a REST api, you could build any UI you want on top of that... On Tue, Oct 6, 2015 at 9:37 AM, shahid qadri wrote: > Hi Folks > > How i c

Re: SparkR Error in sparkR.init(master=“local”) in RStudio

Have you built the Spark jars? Can you run the Spark Scala shell? --Hossein On Tuesday, October 6, 2015, Khandeshi, Ami wrote: > > Sys.setenv(SPARKR_SUBMIT_ARGS="--verbose sparkr-shell") > > Sys.setenv(SPARK_PRINT_LAUNCH_COMMAND=1) > > > > sc <- sparkR.init(master="local") > Launching java with

Re: How can I disable logging when running local[*]?

Thanks. Any chance you know how to pass this to a Scala app that is run via TypeSafe activator? I tried putting it $JAVA_OPTS but I get: Unrecognized option: --driver-java-options Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. I tr

API to run spark Jobs

Hi Folks How i can submit my spark app(python) to the cluster without using spark-submit, actually i need to invoke jobs from UI - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h.

Weird performance pattern of Spark Streaming (1.4.1) + direct Kafka

streaming rate. Here are comparable metrics for two successive tasks: *Slow*: ​ Executor IDAddressTask TimeTotal TasksFailed TasksSucceeded Tasks 20151006-044141-2408867082-5050-21047-S0dnode-3.hdfs.private:3686322 s303 20151006-044141-2408867082-5050-21047-S1dnode-0.hdfs.private:4381240 s11011

Trying PCA on spark but serialization is error thrown

Hi, I tried to used the PCA object in one of my project but end up receiving a serialization error. Any help would be appreciated. Example taken from https://spark.apache.org/docs/latest/mllib-feature-extraction.html#pca My Code: val selector = new PCA(20) val transformer = selector.fit(discretiz

Re: ORC files created by Spark job can't be accessed using hive table

Hi Ted thanks I know I solved that by using dataframe for both reading and writing. I am running into different problem now if spark can read hive orc files why can't hive read orc files created by Spark? On Oct 6, 2015 9:28 PM, "Ted Yu" wrote: > See this thread: > http://search-hadoop.com/m/q3RT

Spark 1.3.1 on Yarn not using all given capacity

Hi, This post might be a duplicate with updates from another one (by me), sorry in advance I have an HDP 2.3 cluster running Spark 1.3.1 on 6 nodes (edge + master + 4 workers) Each worker has 8 cores and 40G of RAM available in Yarn That makes a total of 160GB and 32 cores I'm running a job

Re: ORC files created by Spark job can't be accessed using hive table

See this thread: http://search-hadoop.com/m/q3RTtwwjNxXvPEe1 A brief search in Spark JIRAs didn't find anything opened on this subject. On Tue, Oct 6, 2015 at 8:51 AM, unk1102 wrote: > Hi I have a spark job which creates ORC files in partitions using the > following code > > > dataFrame.write()

ORC files created by Spark job can't be accessed using hive table

Hi I have a spark job which creates ORC files in partitions using the following code dataFrame.write().mode(SaveMode.Append).partitionBy("entity","date").format("orc").save("baseTable"); Above code creates successfully orc files which is readable in Spark dataframe But when I try to load orc f

RE: SparkR Error in sparkR.init(master=“local”) in RStudio

> Sys.setenv(SPARKR_SUBMIT_ARGS="--verbose sparkr-shell") > Sys.setenv(SPARK_PRINT_LAUNCH_COMMAND=1) > > sc <- sparkR.init(master="local") Launching java with spark-submit command /C/DevTools/spark-1.5.1/bin/spark-submit.cmd --verbose sparkr-shell C:\Users\a554719\AppData\Local\Temp\Rtmpw11KJ

Re: spark-submit --packages using different resolver

Hi Burak, Thank you for the tip. Unfortunately it does not work. It throws: java.net.MalformedURLException: unknown protocol: s3n] at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1003) at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSub

Re: [Spark on YARN] Multiple Auxiliary Shuffle Service Versions

On 6 Oct 2015, at 01:23, Andrew Or mailto:and...@databricks.com>> wrote: Both the history server and the shuffle service are backward compatible, but not forward compatible. This means as long as you have the latest version of history server / shuffle service running in your cluster then you'r

Trying PCA on spark but serialization is error thrown

Hi, I tried to used the PCA object in one of my project but end up receiving a serialization error. Any help would be appreciated. Example taken from https://spark.apache.org/docs/latest/mllib-feature-extraction.html#pca My Code: val selector = new PCA(20) val transformer = selector.fit(discretiz

Re: does KafkaCluster can be public ?

For what it's worth, I also use this class in an app, but it happens to be from Java code where it acts as if it's public. So no problem for my use case, but I suppose, another small vote for the usefulness of this class to the caller. I end up using getLatestLeaderOffsets to figure out how to init

Re: Broadcast var is null

Yes, see https://issues.apache.org/jira/browse/SPARK-4170 The reason was kind of complicated, and the 'fix' was just to warn you against subclassing App! yes, use a main() method. On Tue, Oct 6, 2015 at 3:15 PM, Nick Peterson wrote: > This might seem silly, but... > > Stop having your object ex

Re: does KafkaCluster can be public ?

Or maybe annotate with @DeveloperApi Cheers On Tue, Oct 6, 2015 at 7:24 AM, Cody Koeninger wrote: > I personally think KafkaCluster (or the equivalent) should be made > public. When I'm deploying spark I just sed out the private[spark] and > rebuild. > > There's a general reluctance to make th

Re: does KafkaCluster can be public ?

I personally think KafkaCluster (or the equivalent) should be made public. When I'm deploying spark I just sed out the private[spark] and rebuild. There's a general reluctance to make things public due to backwards compatibility, but if enough people ask for it... ? On Tue, Oct 6, 2015 at 6:51 AM

Re: Broadcast var is null

This might seem silly, but... Stop having your object extend App, and instead give it a main method. That's worked for me recently when I've had this issue. (There was a very old issue in Spark related to this; it would seem like a possible regression, if this fixes it for you.) -- Nick On Tue,

Re: compatibility issue with Jersey2

Maybe build Spark with -Djersey.version=2.9 ? Cheers On Tue, Oct 6, 2015 at 5:57 AM, oggie wrote: > I have some jersey compatibility issues when I tried to upgrade from 1.3.1 > to > 1.4.1.. > > We have a Java app written with spark 1.3.1. That app also uses Jersey 2.9 > client to make external

RE: SparkR Error in sparkR.init(master=“local”) in RStudio

What you have done is supposed to work. Need more debugging information to find the cause. Could you add the following lines before calling sparkR.init()? Sys.setenv(SPARKR_SUBMIT_ARGS="--verbose sparkr-shell") Sys.setenv(SPARK_PRINT_LAUNCH_COMMAND=1) Then to see if you can find any hint in t

compatibility issue with Jersey2

I have some jersey compatibility issues when I tried to upgrade from 1.3.1 to 1.4.1.. We have a Java app written with spark 1.3.1. That app also uses Jersey 2.9 client to make external calls. We see spark 1.4.1 uses Jersey 1.9. In 1.3.1 we were able to add some exclusions to our pom and everythi

Re: Broadcast var is null

I've reduced the code to the code below - no streaming, no Kafka, no checkpoint. Unfortunately the end result is the same - "broadcastVar is null" printed in the worker log. Any suggestion on what I'm missing would be very much appreciated ! object BroadcastTest extends App { val logger = Logge

Re: SparkR Error in sparkR.init(master=“local”) in RStudio

I couldn't get this working... I have have JAVA_HOME set. I have defined SPARK_HOME Sys.setenv(SPARK_HOME="c:\DevTools\spark-1.5.1") .libPaths(c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"), .libPaths())) library("SparkR", lib.loc="c:\\DevTools\\spark-1.5.1\\lib") library(SparkR) sc<-sparkR.ini

Re: extracting the top 100 values from an rdd and save it as text file

Hello patelmiteshn, This could do the trick : rdd1 = rdd.sortBy(lambda x: x[1], ascending=False) rdd2 = rdd1.zipWithIndex().filter(tuple => tuple._2 < 1) rdd2.saveAsTextFile() -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/extracting-the-top-100-val

RE: Graphx hangs and crashes on EdgeRDD creation

Hi, I get the same problem with both the CanonicalVertexCut and RandomVertexCut, with the graph code as follows val graph = Graph.fromEdgeTuples(indexedEdges, 0, None, StorageLevel.MEMORY_AND_DISK_SER, StorageLevel.MEMORY_AND_DISK_SER); graph.partitionBy(PartitionStrategy.RandomVertexCut); graph

Enabling kryo serialization slows down machine learning app.

Hi, my team is setting up a machine-learning framework based on Spark's mlib, that currently uses logistic regression. I enabled Kryo serialization and enforced class registration, so I know that all the serialized classes are registered. However, the running times when Kryo serialization is enable

Re: does KafkaCluster can be public ?

You can put a class in the org.apache.spark namespace to access anything that is private[spark]. You can then make enrichments there to access whatever you need. Just beware upgrade pain :) El martes, 6 de octubre de 2015, Erwan ALLAIN escribió: > Hello, > > I'm currently testing spark streaming

Re: [Spark on YARN] Multiple Auxiliary Shuffle Service Versions

Hi Andrew, thanks a lot for the clarification! Regards, Andreas On Tue, Oct 6, 2015 at 2:23 AM, Andrew Or wrote: > Hi all, > > Both the history server and the shuffle service are backward compatible, > but not forward compatible. This means as long as you have the latest > version of history s

RE: DStream Transformation to save JSON in Cassandra 2.1

Thanks Ashish and Jean. I am using scala API, so I used the case class case class Coordinate(id: String, ax: Double, ay: Double, az: Double, oa: Double, ob: Double, og:Double) def mapToCoordinate(jsonMap: Map[String,Any]): Coordinate = { //map the coordinate } val lines = KafkaUtils.createStr

Re: Spark thrift service and Hive impersonation.

On 5 Oct 2015, at 22:51, Jagat Singh mailto:jagatsi...@gmail.com>> wrote: Hello Steve, Thanks for confirmation. Is there any work planned work on this. Not that I'm aware of, though somebody may be doing it. SparkSQL is not hive. It uses some of the libraries -the thriftserver stuff is sub

GenericMutableRow and Row mismatch on Spark 1.5?

Hi Guys, I'm upgrading to Spark 1.5. In our previous version (Spark 1.3 but it was OK on 1.4 as well) we created GenericMutableRow (org.apache.spark.sql.catalyst.expressions.GenericMutableRow) and return it as org.apache.spark.sql.Row Starting from Spark 1.5 GenericMutableRow isn't extends Row.

unresolved dependency: org.apache.spark#spark-streaming_2.10;1.5.0: not found

Hi, I am trying to use Spark 1.5, Mlib, but I keep getting "sbt.ResolveException: unresolved dependency: org.apache.spark#spark-streaming_2.10;1.5.0: not found" . It is weird that this happens, but I could not find any solution for this. Does any one faced the same issue? best, /Shahab Here i

does KafkaCluster can be public ?

Hello, I'm currently testing spark streaming with kafka. I'm creating DirectStream with KafkaUtils and everything's fine. However I would like to use the signature where I can specify my own message handler (to play with partition and offset). In this case, I need to manage offset/partition by mys

Re: GraphX: How can I tell if 2 nodes are connected?

Ok, thanks, just wanted to make sure I wasn't missing something obvious. I've worked with Neo4j cypher as well, where it was rather more obvious. e.g. http://neo4j.com/docs/milestone/query-match.html#_shortest_path http://neo4j.com/docs/stable/cypher-refcard/ Dino. On 6 October 2015 at 06:43, Ro

Re: StructType has more rows, than corresponding Row has objects.

Davies, that seemed to be my issue, my colleague helped me to resolved it. The problem was that we build RDD and corresponding StructType by ourselves (no json, parquet, cassandra, etc - we take a list of business objects and convert them to Rows, then infer struct type) and I missed one thing. --

Re: Spark SQL with Hive error: "Conf non-local session path expected to be non-null;"

I have recently upgraded from spark 1.2 to spark 1.3. After upgrade I made necessary changes to incorporate DataFrames instead of JavaSchemaRDD. Now I am getting this error message *("org.apache.spark.sql.AnalysisException: Conf non-local session path expected to be non-null")* while running my sp