Why does Apache Spark Master shutdown when Zookeeper expires the session

As I understand, Apache Spark Master can be run in high availability mode using Zookeeper. That is, multiple Spark masters can run in Leader/Follower mode and these modes are registered with Zookeeper. In our scenario Zookeeper is expiring the Spark Master's session which is acting as Leader. So

Re: Spark 2.4.0 Master going down

Hi Akshay Thanks for the response please find below the answers to your questions. 1. We are running Spark in cluster mode the cluster manager being Spark's standalone cluster manager. 2. All the ports are open and we preconfigure on what ports the communication should happen and modify firewall

Re: Spark 2.4.0 Master going down

Hi Akshay Thanks for the response please find below the answers to your questions. 1. We are running Spark in cluster mode the cluster manager being Spark's standalone cluster manager. 2. All the ports are open and we preconfigure on what ports the communication should happen and modify firewall

Spark 2.4.0 Master going down

Hi All We are running Spark version 2.4.0 and we run few Spark streaming jobs listening on Kafka topics. We receive an average of 10-20 msgs per second. And the Spark master has been going down after 1-2 hours of it running. Exception is given below: Along with that spark executors also get

Spark streaming rawSocketStream with protobuf

I am trying the spark streaming and listening to a socket, I am using the rawSocketStream method to create a receiver and a DStream. But when I print the DStream I get the below exception.*Code to create a DStream:*JavaSparkContext jsc = new JavaSparkContext("Master", "app");JavaStreamingContext

Twitter receiver not running in spark 1.6.0

Hi forum For some reason if I include a twitter receiver and start the streaming context, I get the below exception not sure why Can someone let me know if anyone has already encountered this issue or am I doing something wrong? java.lang.ArithmeticException: / by zero at

Re: How to contribute by picking up starter bugs

Thanks a lot Jim, looking to forward to pick up some bugs. On Mon, Dec 28, 2015 at 8:42 AM, jiml [via Apache Spark User List] < ml-node+s1001560n25813...@n3.nabble.com> wrote: > You probably want to start on the dev list: > http://apache-spark-developers-list.1001551.n3.nabble.com/ > > I have

How to contribute by picking up starter bugs

Hi >From the how to contribute page of spark jira project I came to know that I can start by picking up the starter label bugs. But who will assign me these bugs? Or should I just fix them and create a pull request. Will be glad to help the project. -- View this message in context:

Spark 1.4.0 org.apache.spark.sql.AnalysisException: cannot resolve 'probability' given input columns

Hi forumI am currently using Spark 1.4.0, and started using the ML pipeline framework.I ran the example program ml.JavaSimpleTextClassificationPipeline which uses the LogisticRegression. But I wanted to do multiclass classification, so I used DecisionTreeClassifier present in the

Spark 1.4.0 compute-classpath.sh

Hi forum I have downloaded the latest spark version 1.4.0 and started using it. But I couldn't find the compute-classpath.sh file in bin/ which I am using in previous versions to provide third party libraries to my application. Can anyone please let me know where I can provide CLASSPATH with my

Spark streaming - textFileStream/fileStream - Get file name

Hi Forum, Using spark streaming and listening to the files in HDFS using textFileStream/fileStream methods, how do we get the fileNames which are read by these methods? I used textFileStream which has file contents in JavaDStream and I got no success with fileStream as it is throwing me a

Spark 1.3.1 JavaStreamingContext - fileStream compile error

Hi Forum I am facing below compile error when using the fileStream method of the JavaStreamingContext class. I have copied the code from JavaAPISuite.java test class of spark test code. The error message is

Spark 1.3.1 JavaStreamingContext - fileStream compile error

Hi Forum I am facing below compile error when using the fileStream method of the JavaStreamingContext class. I have copied the code from JavaAPISuite.java test class of spark test code. Please help me to find a solution for this.

Re: java.io.InvalidClassException: org.apache.spark.api.java.JavaUtils$SerializableMapWrapper; no valid constructor

The workaround was to wrap the map returned by spark libraries into HashMap and then broadcast them. Could anyone please let me know if there is any issue open? -- View this message in context:

java.io.InvalidClassException: org.apache.spark.api.java.JavaUtils$SerializableMapWrapper; no valid constructor

Hi forum, We have been using spark 1.1.0 and due to some bugs in it, we upgraded to latest 1.3.0 from the master branch. And we are getting the below error while using the broadcast variable. Could anyone please point out whats wrong here? org.apache.spark.SparkException: Job aborted due to

java.io.InvalidClassException: org.apache.spark.api.java.JavaUtils$SerializableMapWrapper; no valid constructor

Hi forum, We have been using spark 1.1.0 and due to some bugs in it, we upgraded to latest 1.3.0 from the master branch. And we are getting the below error while using the broadcast variable. Could anyone please point out whats wrong here? org.apache.spark.SparkException: Job aborted due to

Multiple SparkContexts in same Driver JVM

Hi Forum, Is it not possible to run multiple SparkContexts concurrently without stopping the other one in the spark 1.3.0. I have been trying this out and getting the below error. Caused by: org.apache.spark.SparkException: Only one SparkContext may be running in this JVM (see SPARK-2243). To

Re: Issue with Spark latest 1.2.0 build - ClassCastException from [B to SerializableWritable

The above issue happens while trying to do the below activity on JavaRDD (calling take() on rdd) JavaRDDString loadedRDD = sc.textFile(...); String[] tokens = loadedRDD.take(1).get(0).split(,); -- View this message in context:

Re: Issue with Spark latest 1.2.0 build - ClassCastException from [B to SerializableWritable

Hi Sean Thanks for reply, We upgraded our spark cluster from 1.1.0 to 1.2.0. And we also thought that this issue might be due to mis matching spark jar versions. But we double checked and re installed our app completely in a new system with spark-1.2.0 distro, but still no result. Facing the same

Issue with Spark latest 1.2.0 build - ClassCastException from [B to SerializableWritable

Hello forum, We are using spark distro built from the source of latest 1.2.0 tag. And we are facing the below issue, while trying to act upon the JavaRDD instance, the stacktrace is given below. Can anyone please let me know, what can be wrong here? java.lang.ClassCastException: [B cannot be

Issue with Spark latest 1.2.0 build - ClassCastException from [B to SerializableWritable

Hello forum, We are using spark distro built from the source of latest 1.2.0 tag. And we are facing the below issue, while trying to act upon the JavaRDD instance, the stacktrace is given below. Can anyone please let me know, what can be wrong here? java.lang.ClassCastException: [B cannot be

stdout in spark applications

Hi Forum, I am running a simple spark application in 1 master and 1 worker. Submitting my application through spark submit as a java program. I have sysout in the program, but I am not finding these sysouts in stdout/stderr links in web ui of master as well in the SPARK_HOME/work directory.

Re: stdout in spark applications

Got my answer from this thread, http://apache-spark-user-list.1001560.n3.nabble.com/no-stdout-output-from-worker-td2437.html -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/stdout-in-spark-applications-tp18056p18134.html Sent from the Apache Spark User List

Re: Spark MLLIB Decision Tree - ArrayIndexOutOfBounds Exception

Hi Joseph, Thanks for the help. I have tried this DecisionTree example with the latest spark code and it is working fine now. But how do we choose the maxBins for this model? Thanks Lokesh -- View this message in context:

Spark MLLIB Decision Tree - ArrayIndexOutOfBounds Exception

Hi All, I am trying to run the spark example JavaDecisionTree code using some external data set. It works for certain dataset only with specific maxBins and maxDepth settings. Even for a working dataset if I add a new data item I get a ArrayIndexOutOfBounds Exception, I get the same exception

Re: Spark MLLIB Decision Tree - ArrayIndexOutOfBounds Exception

Hi Joseph I am using spark 1.1.0 the latest version, I will try to update to the current master and check. The example I am running is JavaDecisionTree, the dataset is of libsvm format containing 1. 45 instances of training sample. 2. 5 features 3. I am not sure what is feature type, but