I’m trying to submit a stand-alone Flink job to a YARN cluster running on EMR (Elastic MapReduce) nodes in AWS. When it tries to start a container for the Job Manager, it fails. The error message from the container is below. The command I’m using is:
$ flink run -m yarn-cluster -yn 1 -ynm test1:WordCount ./flink-1.0.3/examples/streaming/WordCount.jar I have tried adding log4j and slf4j libraries to the classpath using -C and that doesn’t help. This does not happen on other YARN clusters I have that are not EMR nodes. And it doesn’t happen on my EMR cluster if I use "yarn-session.sh" to create a Flink cluster in the YARN cluster and then use “flink run …” to submit the job to the Flink cluster. Does anyone out there know how I could fix this? Thanks in advance for any help you can give. Error message in the jobmanager.err file: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/log4j/Level at org.apache.hadoop.mapred.JobConf.<clinit>(JobConf.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:278) at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:95) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136) at org.apache.hadoop.security.Groups.<init>(Groups.java:79) at org.apache.hadoop.security.Groups.<init>(Groups.java:74) at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:303) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:283) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260) at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:790) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633) at org.apache.flink.yarn.ApplicationMasterBase.run(ApplicationMasterBase.scala:64) at org.apache.flink.yarn.ApplicationMaster$.main(ApplicationMaster.scala:36) at org.apache.flink.yarn.ApplicationMaster.main(ApplicationMaster.scala) Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Level at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 18 more [cid:01830D03-C3D5-4C98-BB4D-27DC3889EBFC] Bruce Hanson Software Engineer HERE Predictive Analytics HERE Seattle 701 Pike St, Suite 2000, Seattle, WA 98101 47° 36' 41" N. 122° 19' 57" W<http://here.com/usa/seattle/98101/pike-st/701/map=47.611439,-122.332741,17/title=HERE%20Seattle%20-%20701%20Pike%20Street> [cid:822C9F40-B358-441C-86A3-270D8B84D3CC]<http://360.here.com/> [cid:8C181B0D-7AD1-4C78-9D87-EB8B9235DE19] <https://twitter.com/here> [cid:6EF6C991-FACA-474E-8D17-3C91F7BD47C1] <https://www.facebook.com/here> [cid:53CE4C6B-3D8C-4E38-AE94-C7B2006A2BCE] <https://linkedin.com/company/heremaps> [cid:EA73988E-374C-4D16-A59C-AF82144FC71C] <https://www.instagram.com/here>
