Yeah, ran it on yarn-cluster mode.
On Tue, Aug 5, 2014 at 12:17 PM, Akhil Das <ak...@sigmoidanalytics.com> wrote: > Are you sure that you were not running SparkPi in local mode? > > Thanks > Best Regards > > > On Wed, Aug 6, 2014 at 12:43 AM, Sunny Khatri <sunny.k...@gmail.com> > wrote: > >> Well I was able to run the SparkPi, that also does the similar stuff, >> successfully. >> >> >> On Tue, Aug 5, 2014 at 11:52 AM, Akhil Das <ak...@sigmoidanalytics.com> >> wrote: >> >>> For that UI to have some values, your process should do some operation. >>> Which is not happening here ( 14/08/05 18:03:13 WARN >>> YarnClusterScheduler: Initial job has not accepted any resources; check >>> your cluster UI to ensure that workers are registered and have sufficient >>> memory ) >>> >>> Can you open up a spark-shell and try some simple code? ( *val x = >>> sc.parallelize(1 to 1000000).filter(_<100).collect()* ) >>> >>> Just to make sure your cluster setup is proper and is working. >>> >>> Thanks >>> Best Regards >>> >>> >>> On Wed, Aug 6, 2014 at 12:17 AM, Sunny Khatri <sunny.k...@gmail.com> >>> wrote: >>> >>>> The only UI I have currently is the Application Master (Cluster mode), >>>> with the following executor nodes status: >>>> Executors (3) >>>> >>>> - *Memory:* 0.0 B Used (3.7 GB Total) >>>> - *Disk:* 0.0 B Used >>>> >>>> Executor IDAddress RDD BlocksMemory Used Disk UsedActive Tasks Failed >>>> TasksComplete Tasks Total TasksTask Time Shuffle ReadShuffle Write 1 >>>> <add1> 0 0.0 B / 1766.4 MB 0.0 B 0 0 0 0 0 ms 0.0 B 0.0 B 2<add2> 0 >>>> 0.0 B / 1766.4 MB 0.0 B0 0 00 0 ms0.0 B 0.0 B <driver> <add3> 0 0.0 B >>>> / 294.6 MB 0.0 B 0 0 0 0 0 ms 0.0 B 0.0 B >>>> >>>> >>>> On Tue, Aug 5, 2014 at 11:32 AM, Akhil Das <ak...@sigmoidanalytics.com> >>>> wrote: >>>> >>>>> Are you able to see the job on the WebUI (8080)? If yes, how much >>>>> memory are you seeing there specifically for this job? >>>>> >>>>> [image: Inline image 1] >>>>> >>>>> Here you can see i have 11.8Gb RAM on both workers and my app is using >>>>> 11GB. >>>>> >>>>> 1. What are all the memory that you are seeing in your case? >>>>> 2. Make sure your application is using the same spark URI (as seen in >>>>> the top left of the webUI) while creating the SparkContext. >>>>> >>>>> >>>>> >>>>> Thanks >>>>> Best Regards >>>>> >>>>> >>>>> On Tue, Aug 5, 2014 at 11:38 PM, Sunny Khatri <sunny.k...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I'm trying to run a spark application with the executor-memory 3G. >>>>>> but I'm running into the following error: >>>>>> >>>>>> 14/08/05 18:02:58 INFO DAGScheduler: Submitting Stage 0 (MappedRDD[5] at >>>>>> map at KMeans.scala:123), which has no missing parents >>>>>> 14/08/05 18:02:58 INFO DAGScheduler: Submitting 1 missing tasks from >>>>>> Stage 0 (MappedRDD[5] at map at KMeans.scala:123) >>>>>> 14/08/05 18:02:58 INFO YarnClusterScheduler: Adding task set 0.0 with 1 >>>>>> tasks >>>>>> 14/08/05 18:02:59 INFO CoarseGrainedSchedulerBackend: Registered >>>>>> executor: >>>>>> Actor[akka.tcp://sparkexecu...@test-hadoop2.vpc.natero.com:54358/user/Executor#1670455157] >>>>>> with ID 2 >>>>>> 14/08/05 18:02:59 INFO BlockManagerInfo: Registering block manager >>>>>> test-hadoop2.vpc.natero.com:39156 with 1766.4 MB RAM >>>>>> 14/08/05 18:03:13 WARN YarnClusterScheduler: Initial job has not >>>>>> accepted any resources; check your cluster UI to ensure that workers are >>>>>> registered and have sufficient memory >>>>>> 14/08/05 18:03:28 WARN YarnClusterScheduler: Initial job has not >>>>>> accepted any resources; check your cluster UI to ensure that workers are >>>>>> registered and have sufficient memory >>>>>> 14/08/05 18:03:43 WARN YarnClusterScheduler: Initial job has not >>>>>> accepted any resources; check your cluster UI to ensure that workers are >>>>>> registered and have sufficient memory >>>>>> 14/08/05 18:03:58 WARN YarnClusterScheduler: Initial job has not >>>>>> accepted any resources; check your cluster UI to ensure that workers are >>>>>> registered and have sufficient memory >>>>>> >>>>>> >>>>>> Tried tweaking executor-memory as well, but same result. It always gets >>>>>> stuck registering the block manager. >>>>>> >>>>>> >>>>>> Are there any other settings that needs to be adjusted. >>>>>> >>>>>> >>>>>> Thanks >>>>>> >>>>>> Sunny >>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
