Re: Spark and Kafka direct approach problem

NoSuchMethodError usually appears because of a difference in the library versions. Check the version of the libraries you downloaded, the version of spark, the version of Kafka. On 4 May 2016 at 16:18, Luca Ferrari wrote: > Hi, > > I’m new on Apache Spark and I’m trying

Re: Spark Streaming Job get killed after running for about 1 hour

I am using the latest Spark version 1.6 I have increased the maximum number of open files using this command *sysctl -w fs.file-max=3275782* Also I increased the limit for the user who run the spark job by updating the /etc/security/limits.conf file. Soft limit is 1024 and Hard limit is 65536.

Re: Sending events to Kafka from spark job

Dear Andy, As far as I understand, the transformations are applied to the RDDs not to the data and I need to send the actual data to Kafka. This way, I think I should perform at least one action to make spark load the data. Kindly correct me if I do not understand this the correct way. Best

Re: Apache Spark data locality when integrating with Kafka

Diwakar We have our own servers. We will not use any cloud service like Amazon's On 7 February 2016 at 18:24, Diwakar Dhanuskodi < diwakar.dhanusk...@gmail.com> wrote: > Fanoos, > Where you want the solution to be deployed ?. On premise or cloud? > > Regards > Diwakar . > > > > Sent from

Re: java.lang.ClassNotFoundException: org.apache.spark.streaming.twitter.TwitterReceiver

are supposed to submit the fat-jar of the application that include the > spark-streaming-twitter dependency (and its subdeps) but not > spark-streaming and spark-core. > > On Mon, Nov 9, 2015 at 1:02 AM, أنس الليثي <dev.fano...@gmail.com> wrote: > >> I tried to remove mav

Re: java.lang.ClassNotFoundException: org.apache.spark.streaming.twitter.TwitterReceiver

tried to change eclipse with another version, but the same problem exist. :( :( :( :( On 9 November 2015 at 10:47, أنس الليثي <dev.fano...@gmail.com> wrote: > I tried both, the same exception still thrown > > On 9 November 2015 at 10:45, Sean Owen <so...@cloudera.com> wrote: &

Spark Slave always fails to connect to master

I am trying to install a standalone spark cluster. I prepared 3 virtual machines each with Ubuntu installed. The three machines consists a cluster with one master and two slaves. I followed the steps in the documentation of Apache spark. I started the master script from the master node and it

Re: Connection PHP application to Spark Sql thrift server

Sorry for late reply. I have tried to connect to the hive server instead of the spark sql but the same exception is thrown in the hive server logs. The only difference is the hive log has a little more information than the spark sql logs. The hive server logs has this message TTransportException

Re: Connecting a PHP/Java applications to Spark SQL Thrift Server

Thanks very much, I used it and works fine with me. On 4 March 2015 at 11:56, Arush Kharbanda ar...@sigmoidanalytics.com wrote: For java You can use hive-jdbc connectivity jars to connect to Spark-SQL. The driver is inside the hive-jdbc Jar.

Re: InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag

Hadoop version : 2.6.0 Spark Version : 1.2.1 here is also the pom.xml project xmlns=http://maven.apache.org/POM/4.0.0; xmlns:xsi= http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation= http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;