Problem solved. Thank you for your help. @Ted Yu Other issues where my mistakes. I have a dedicated script which updates/builds/"deploys" YARN from sources. I was starting NN with the "-upgrade" option which unsynchronized NN version, also leading to broken DN. Quick NN format and deletion of DN data solved the issue (I am working on a sandbox cluster, so that is not a problem). I have modified the script to start the NN without the upgrade option.
Two quick questions: - when should I use the NN upgrade option, should it be only used to upgrade NN between new version (example 2.0.4 -> 2.0.5). How can I automate this process? - Is the "design/functionality" of my magical script correct? How can I avoid future problems like the solved one? YARN update script - update src to trunk (opt) - package YARN - build dist (tar ball) - unpack new_dist - overwrite new_dist conf with prev_dist conf (this has lead to the problem with DefaultResourceCalculator, my conf is bare minimum to work in pseudo-distributed mode) - start YARN For any suggestions I would be grateful. regards tmp 2013/8/12 Ted Yu <[email protected]> > Can you check the config entry > for yarn.scheduler.capacity.resource-calculator ? > It should point > to org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator > > bq. I was able to fix all issues > > What other issues came up ? > > Thanks > > > On Sun, Aug 11, 2013 at 2:07 PM, Rob Blah <[email protected]> wrote: > >> Hi again >> >> From a little investigation I have performed I have observed the >> following. I assume the module responsible for this class is >> hadoop-yarn-common. >> >> During RM init it crashes since it is looking for a class >> DefaultResourceCalculator in >> org.apache.hadoop.yarn.server.resourcemanager.resource.DefaultResourceCalculator, >> while the class is present in hadoop-yarn-common-3.0.0-SNAPSHOT.jar but >> under org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator. Thus >> the RM crashes. Anybody knows how I can fix this? I would very grateful for >> any help. >> >> regards >> tmp >> >> >> 2013/8/11 Rob Blah <[email protected]> >> >>> Hi >>> >>> I have a strange problem, regarding missing class, the >>> DefaultResourceCalculator. I have a single node sandbox cluster working in >>> a pseudo-distributed mode. The cluster was working fine yesterday, however >>> today it stopped working. I was able to fix all issues except the following >>> problem in ResourceManager: >>> 2013-08-11 12:12:42,425 FATAL >>> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error >>> starting ResourceManager >>> java.lang.RuntimeException: java.lang.RuntimeException: >>> java.lang.ClassNotFoundException: Class >>> org.apache.hadoop.yarn.server.resourcemanager.resource.DefaultResourceCalculator >>> not found >>> at >>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1753) >>> at >>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getResourceCalculator(CapacitySchedulerConfiguration.java:333) >>> at >>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:258) >>> at >>> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:241) >>> at >>> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) >>> at >>> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:826) >>> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: >>> Class >>> org.apache.hadoop.yarn.server.resourcemanager.resource.DefaultResourceCalculator >>> not found >>> at >>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1721) >>> at >>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1745) >>> ... 5 more >>> Caused by: java.lang.ClassNotFoundException: Class >>> org.apache.hadoop.yarn.server.resourcemanager.resource.DefaultResourceCalculator >>> not found >>> at >>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1625) >>> at >>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1719) >>> ... 6 more >>> 2013-08-11 12:12:42,426 INFO >>> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: >>> SHUTDOWN_MSG: >>> >>> I build YARN from sources, daily updated to the newest revision in >>> trunk. The class DefaultResourceCalculator exists and is present in YARN's >>> sources. I am using (currently) trunk revision 1512895. I build YARN >>> project with the following command: >>> mvn clean package -Pdist -Dtar -DskipTests >>> I create tar ball with the use of script provided in the sources: >>> dist-tar-stitching.sh (hadoop-dist/target) >>> >>> regards >>> tmp >>> >> >> >
