Got it to work...thanks a lot for the help! I started a new cluster where
Spark has Yarn as a dependency. I ran it with the script with local[2] and
it worked (this same script did not work with Spark in standalone mode).
A follow up question...I have seen this question posted around the internet
I thought I was running it in local mode as
http://spark.apache.org/docs/1.1.1/submitting-applications.html says that
if I don't include "--deploy-mode cluster" then it will run as local mode?
I tried both of the scripts above and they gave the same result as the
script I was running before.
Also
You don't submit it like that :/
You use [*] things when you run the job in local mode, whereas here you are
running it in stand alone cluster mode.
You can try either of these:
1.
/opt/cloudera/parcels/CDH-5.2.1-1.cdh5.2.10.12/lib/spark/bin/spark-submit
--class SimpleApp --master spark://10.0.1
I tried submitting the application like this with 2 cores as you can see
with the [2].
/opt/cloudera/parcels/CDH-5.2.1-1.cdh5.2.10.12/lib/spark/bin/spark-submit
--class SimpleApp --master spark://10.0.1.230:7077[2] --jars $(echo
/home/ec2-user/sparkApps/SimpleApp/lib/*.jar | tr ' ' ',')
/home/ec2
How many cores are you allocated/seeing in the webui? (that usually runs on
8080, for cloudera i think its 18080). Most likely the job is being
allocated 1 core (should be >= 2 cores) and that's why the count is never
happening.
Thanks
Best Regards
On Mon, Dec 29, 2014 at 2:22 PM, Suhas Shekar w
So it got rid of the logs, but the problem still persists that :
a) The program never terminates (I have pasted all output after the Hello
World statements below)
b) I am not seeing the word count
c) I tried adding [2] next to my 10.0.1.232:2181 looking at this post
http://apache-spark-user-list
Now, Add these lines to get ride of those logs
import org.apache.log4j.Logger
import org.apache.log4j.Level
Logger.getLogger("org").setLevel(Level.OFF)
Logger.getLogger("akka").setLevel(Level.OFF)
Thanks
Best Regards
On Mon, Dec 29, 2014 at 2:09 PM, Suhas Shekar wrote:
> Hmmm
Hmmm..soo I added 1 (10,000) to jssc.awaitTermination , however it does
not stop. When I am not pushing in any data it gives me this:
14/12/29 08:35:12 INFO ReceiverTracker: Stream 0 received 0 blocks
14/12/29 08:35:12 INFO JobScheduler: Added jobs for time 1419860112000 ms
14/12/29 08:35:14 I
If you want to stop the streaming after 10 seconds, then use
ssc.awaitTermination(1). Make sure you push some data to kafka for the
streaming to consume within the 10 seconds.
Thanks
Best Regards
On Mon, Dec 29, 2014 at 1:53 PM, Suhas Shekar wrote:
> I'm very close! So I added that and then
I'm very close! So I added that and then I added this:
http://mvnrepository.com/artifact/org.apache.kafka/kafka-clients/0.8.2-beta
and it seems as though the stream is working as it says Stream 0 received 1
or 2 blocks as I enter in messages on my kafka producer. However, the
Receiver seems to kee
Add this jar in the dependency
http://mvnrepository.com/artifact/com.yammer.metrics/metrics-core/2.2.0
Thanks
Best Regards
On Mon, Dec 29, 2014 at 1:31 PM, Suhas Shekar wrote:
> Hello Akhil,
>
> I chanced my Kafka dependency to 2.10 (which is the version of kafka that
> was on 10.0.1.232). I am
Hello Akhil,
I chanced my Kafka dependency to 2.10 (which is the version of kafka that
was on 10.0.1.232). I am getting a slightly different error, but at the
same place as the previous error (pasted below).
FYI, when I make these changes to the pom file, I do "mvn clean package"
then cp the new
I made both versions 1.1.1 and I got the same error. I then tried making
both 1.1.0 as that is the version of my Spark Core, but I got the same
error.
I noticed my Kafka dependency is for scala 2.9.2, while my spark streaming
kafka dependency is 2.10.x...I will try changing that next, but don't th
Just looked at the pom file that you are using, why are you having
different versions in it?
org.apache.spark
spark-streaming-kafka_2.10
*1.1.1*
org.apache.spark
spark-streaming_2.10
*1.0.2*
can you make both the versions the same?
Thanks
Best Regards
On Mon, Dec 29, 2014 at 12:44 PM, Su
1) Could you please clarify on what you mean by checking the Scala version
is correct? In my pom.xml file it is 2.10.4 (which is the same as when I
start spark-shell).
2) The spark master URL is definitely correct as I have run other apps with
the same script that use Spark (like a word count with
Make sure you verify the following:
- Scala version : I think the correct version would be 2.10.x
- SparkMasterURL: Be sure that you copied the one displayed on the webui's
top left corner (running on port 8080)
Thanks
Best Regards
On Mon, Dec 29, 2014 at 12:26 PM, suhshekar52 wrote:
> Hello E
16 matches
Mail list logo
