Are you sure that you were not running SparkPi in local mode? Thanks Best Regards
On Wed, Aug 6, 2014 at 12:43 AM, Sunny Khatri <sunny.k...@gmail.com> wrote: > Well I was able to run the SparkPi, that also does the similar stuff, > successfully. > > > On Tue, Aug 5, 2014 at 11:52 AM, Akhil Das <ak...@sigmoidanalytics.com> > wrote: > >> For that UI to have some values, your process should do some operation. >> Which is not happening here ( 14/08/05 18:03:13 WARN >> YarnClusterScheduler: Initial job has not accepted any resources; check >> your cluster UI to ensure that workers are registered and have sufficient >> memory ) >> >> Can you open up a spark-shell and try some simple code? ( *val x = >> sc.parallelize(1 to 1000000).filter(_<100).collect()* ) >> >> Just to make sure your cluster setup is proper and is working. >> >> Thanks >> Best Regards >> >> >> On Wed, Aug 6, 2014 at 12:17 AM, Sunny Khatri <sunny.k...@gmail.com> >> wrote: >> >>> The only UI I have currently is the Application Master (Cluster mode), >>> with the following executor nodes status: >>> Executors (3) >>> >>> - *Memory:* 0.0 B Used (3.7 GB Total) >>> - *Disk:* 0.0 B Used >>> >>> Executor IDAddress RDD BlocksMemory Used Disk UsedActive Tasks Failed >>> TasksComplete Tasks Total TasksTask Time Shuffle ReadShuffle Write 1 >>> <add1> 0 0.0 B / 1766.4 MB 0.0 B 0 0 0 0 0 ms 0.0 B 0.0 B 2<add2> 0 0.0 >>> B / 1766.4 MB 0.0 B0 0 00 0 ms0.0 B 0.0 B <driver> <add3> 0 0.0 B / >>> 294.6 MB 0.0 B 0 0 0 0 0 ms 0.0 B 0.0 B >>> >>> >>> On Tue, Aug 5, 2014 at 11:32 AM, Akhil Das <ak...@sigmoidanalytics.com> >>> wrote: >>> >>>> Are you able to see the job on the WebUI (8080)? If yes, how much >>>> memory are you seeing there specifically for this job? >>>> >>>> [image: Inline image 1] >>>> >>>> Here you can see i have 11.8Gb RAM on both workers and my app is using >>>> 11GB. >>>> >>>> 1. What are all the memory that you are seeing in your case? >>>> 2. Make sure your application is using the same spark URI (as seen in >>>> the top left of the webUI) while creating the SparkContext. >>>> >>>> >>>> >>>> Thanks >>>> Best Regards >>>> >>>> >>>> On Tue, Aug 5, 2014 at 11:38 PM, Sunny Khatri <sunny.k...@gmail.com> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> I'm trying to run a spark application with the executor-memory 3G. but >>>>> I'm running into the following error: >>>>> >>>>> 14/08/05 18:02:58 INFO DAGScheduler: Submitting Stage 0 (MappedRDD[5] at >>>>> map at KMeans.scala:123), which has no missing parents >>>>> 14/08/05 18:02:58 INFO DAGScheduler: Submitting 1 missing tasks from >>>>> Stage 0 (MappedRDD[5] at map at KMeans.scala:123) >>>>> 14/08/05 18:02:58 INFO YarnClusterScheduler: Adding task set 0.0 with 1 >>>>> tasks >>>>> 14/08/05 18:02:59 INFO CoarseGrainedSchedulerBackend: Registered >>>>> executor: >>>>> Actor[akka.tcp://sparkexecu...@test-hadoop2.vpc.natero.com:54358/user/Executor#1670455157] >>>>> with ID 2 >>>>> 14/08/05 18:02:59 INFO BlockManagerInfo: Registering block manager >>>>> test-hadoop2.vpc.natero.com:39156 with 1766.4 MB RAM >>>>> 14/08/05 18:03:13 WARN YarnClusterScheduler: Initial job has not accepted >>>>> any resources; check your cluster UI to ensure that workers are >>>>> registered and have sufficient memory >>>>> 14/08/05 18:03:28 WARN YarnClusterScheduler: Initial job has not accepted >>>>> any resources; check your cluster UI to ensure that workers are >>>>> registered and have sufficient memory >>>>> 14/08/05 18:03:43 WARN YarnClusterScheduler: Initial job has not accepted >>>>> any resources; check your cluster UI to ensure that workers are >>>>> registered and have sufficient memory >>>>> 14/08/05 18:03:58 WARN YarnClusterScheduler: Initial job has not accepted >>>>> any resources; check your cluster UI to ensure that workers are >>>>> registered and have sufficient memory >>>>> >>>>> >>>>> Tried tweaking executor-memory as well, but same result. It always gets >>>>> stuck registering the block manager. >>>>> >>>>> >>>>> Are there any other settings that needs to be adjusted. >>>>> >>>>> >>>>> Thanks >>>>> >>>>> Sunny >>>>> >>>>> >>>>> >>>> >>> >> >
