Re: NetwrokWordCount + Spark standalone
You can open the Master UI running on 8080 port of your ubuntu machine and after submitting the job, you can see how many cores are being used etc from the UI. Thanks Best Regards On Wed, Mar 25, 2015 at 6:50 PM, James King jakwebin...@gmail.com wrote: Thanks Akhil, Yes indeed this is why it works when using local[2] but I'm unclear of why it doesn't work when using standalone daemons? Is there way to check what cores are being seen when running against standalone daemons? I'm running the master and worker on same ubuntu host. The Driver program is running from a windows machine. On ubuntu host command cat /proc/cpuinfo | grep processor | wc -l is giving 2 On Windows machine it is: NumberOfCores=2 NumberOfLogicalProcessors=4 On Wed, Mar 25, 2015 at 2:06 PM, Akhil Das ak...@sigmoidanalytics.com wrote: Spark Streaming requires you to have minimum of 2 cores, 1 for receiving your data and the other for processing. So when you say local[2] it basically initialize 2 threads on your local machine, 1 for receiving data from network and the other for your word count processing. Thanks Best Regards On Wed, Mar 25, 2015 at 6:31 PM, James King jakwebin...@gmail.com wrote: I'm trying to run the Java NetwrokWordCount example against a simple spark standalone runtime of one master and one worker. But it doesn't seem to work, the text entered on the Netcat data server is not being picked up and printed to Eclispe console output. However if I use conf.setMaster(local[2]) it works, the correct text gets picked up and printed to Eclipse console. Any ideas why, any pointers?
NetwrokWordCount + Spark standalone
I'm trying to run the Java NetwrokWordCount example against a simple spark standalone runtime of one master and one worker. But it doesn't seem to work, the text entered on the Netcat data server is not being picked up and printed to Eclispe console output. However if I use conf.setMaster(local[2]) it works, the correct text gets picked up and printed to Eclipse console. Any ideas why, any pointers?
Re: NetwrokWordCount + Spark standalone
Spark Streaming requires you to have minimum of 2 cores, 1 for receiving your data and the other for processing. So when you say local[2] it basically initialize 2 threads on your local machine, 1 for receiving data from network and the other for your word count processing. Thanks Best Regards On Wed, Mar 25, 2015 at 6:31 PM, James King jakwebin...@gmail.com wrote: I'm trying to run the Java NetwrokWordCount example against a simple spark standalone runtime of one master and one worker. But it doesn't seem to work, the text entered on the Netcat data server is not being picked up and printed to Eclispe console output. However if I use conf.setMaster(local[2]) it works, the correct text gets picked up and printed to Eclipse console. Any ideas why, any pointers?
Re: NetwrokWordCount + Spark standalone
Thanks Akhil, Yes indeed this is why it works when using local[2] but I'm unclear of why it doesn't work when using standalone daemons? Is there way to check what cores are being seen when running against standalone daemons? I'm running the master and worker on same ubuntu host. The Driver program is running from a windows machine. On ubuntu host command cat /proc/cpuinfo | grep processor | wc -l is giving 2 On Windows machine it is: NumberOfCores=2 NumberOfLogicalProcessors=4 On Wed, Mar 25, 2015 at 2:06 PM, Akhil Das ak...@sigmoidanalytics.com wrote: Spark Streaming requires you to have minimum of 2 cores, 1 for receiving your data and the other for processing. So when you say local[2] it basically initialize 2 threads on your local machine, 1 for receiving data from network and the other for your word count processing. Thanks Best Regards On Wed, Mar 25, 2015 at 6:31 PM, James King jakwebin...@gmail.com wrote: I'm trying to run the Java NetwrokWordCount example against a simple spark standalone runtime of one master and one worker. But it doesn't seem to work, the text entered on the Netcat data server is not being picked up and printed to Eclispe console output. However if I use conf.setMaster(local[2]) it works, the correct text gets picked up and printed to Eclipse console. Any ideas why, any pointers?
