when the data’s source host is not one of the registered executors, it will also be marked as PROCESS_LOCAL too, though it should have a different NAME for this. I don’t know did someone change this name very recently. but for 0.9, it is the case .
When I say satisfy, yes, if the executors have enough resources to run your tasks. all the tasks will be assigned out in your case for the registered 4 nodes. though the other 16 executors probably will registered in a short while later. but when they got registered, all tasks have already been assigned out. Not sure is this exactly what happened in your cluster. But seems to me likely to be. Your case might not be that executors not run on all spark nodes, but they don’t get registered quick enough. On 2014年10月20日, at 下午2:15, Tao Xiao <xiaotao.cs....@gmail.com> wrote: > Raymond, > > Thank you. > > But I read from other thread that "PROCESS_LOCAL" means the data is in the > same JVM as the code that is running. When data is in the same JVM with the > code that is running, the data should be on the same node as JVM, i.e., the > data can be said to be local. > > Also you said that the tasks will be assigned to available executors which > satisfy the application's requirements. But what requirements must an > executor satisfy so that task can be assigned to it? Do you mean resources > (memory, CPU)? > > Finally, is there any way to guarantee that all executors for an application > will be run on all Spark nodes when data to be processed is big enough (for > example, HBase data resides on all RegionServers) ? > > > > > 2014-10-20 11:35 GMT+08:00 raymond <rgbbones.m...@gmail.com>: > My best guess is the speed that your executors got registered with driver > differs between each run. > > when you run it for the first time, the executors is not fully registered > when task set manager start to assign tasks, and thus the tasks was assigned > to available executors which have already satisfy what you need ,say 86 with > one batch. > > And the “Process_local” does not necessary means that the data is local, it > could be that the executor is not available yet for the data source ( in your > case, might though will be available later). > > If this is the case, you could just sleep a few seconds before run the job. > or there are some patches related and providing other way to sync executors > status before running applications, but I haven’t track the related status > for a while. > > > Raymond > > On 2014年10月20日, at 上午11:22, Tao Xiao <xiaotao.cs....@gmail.com> wrote: > >> Hi all, >> >> I have a Spark-0.9 cluster, which has 16 nodes. >> >> I wrote a Spark application to read data from an HBase table, which has 86 >> regions spreading over 20 RegionServers. >> >> I submitted the Spark app in Spark standalone mode and found that there were >> 86 executors running on just 3 nodes and it took about 30 minutes to read >> data from the table. In this case, I noticed from Spark master UI that >> Locality Level of all executors are "PROCESS_LOCAL". >> >> Later I ran the same app again (without any code changed) and found that >> those 86 executors were running on 16 nodes, and this time it took just 4 >> minutes to read date from the same HBase table. In this case, I noticed that >> Locality Level of most executors are "NODE_LOCAL". >> >> After testing multiple times, I found the two cases above occur randomly. >> >> So I have 2 questions: >> 1) Why would the two cases above occur randomly when I submitted the same >> application multiple times ? >> 2) Would the spread of executors influence locality level ? >> >> Thank you. >> >> > >
