What EMR version are you using? Can you confirm you are using Oozie 5.0? I ask because I encountered a similar problem when we went from Oozie 5.0 to 5.1 while upgrading to EMR 5.22. I believe oozie.jobs.api.generated.xml was added in Oozie 5.1. See https://issues.apache.org/jira/browse/OOZIE-3458, https://issues.apache.org/jira/browse/OOZIE-3457 and related bugs for details. An important point is the code reports "null" when that isn't the real problem.
-Jerry -----Original Message----- From: Daniel Zhang [mailto:java8...@hotmail.com] Sent: Monday, April 1, 2019 6:29 PM To: user@oozie.apache.org Subject: Oozie 5.0 coordinator issue Hi, I am facing a strange issue on Oozie 5.0. I have an Ozzie application that works fine in the workflow mode. Here is the only 2 examples line of my oozie properties file: oozie.wf.application.path=${appBaseDir}/workflow/workflow.xml #oozie.coord.application.path=${appBaseDir}/workflow/coordinator.xml And here is the content of the coordinator.xml file: <coordinator-app name="GoogleAnalytics" start="2018-09-06T23:25Z" end="2100-12-31T23:00Z" frequency="${coord:days(1)}" timezone="UTC" xmlns="uri:oozie:coordinator:0.4"> <action> <workflow> <app-path>${appBaseDir}/workflow/workflow.xml</app-path> <configuration> <property> <name>nameNode</name> <value>${nameNode}</value> </property> <property> <name>jobTracker</name> <value>${jobTracker}</value> </property> </configuration> </workflow> </action> </coordinator-app> Now, this whole thing works on AWS EMR with Oozie 4.3. But with the new version of AWS EMR comes with Oozie 5.0.0, I am facing the following strange error. 1) If I leave the properties file as above to running in the workflow mode by "oozie job -run -config googleanalytics_aws_qa.properties file, everything works under oozie 5.0 2) If I comment out the "wf" line above and uncomment out the "coord" line, I got the following error: [hadoop@ip-10-0-60-152 oozieJobs]$ oozie job -run -config googleanalytics_aws_qa.properties SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/lib/oozie/lib/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/lib/oozie/lib/slf4j-simple-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Error: E0307 : E0307: Runtime error [App directory [null] does not exist and app definition cannot be created because of missing config value [oozie.jobs.api.generated.xml]] And here is the stack trace in the oozie.log: 2019-04-01 22:03:55,806 WARN V1JobsServlet:523 - SERVER[ip-10-0-60-152.ec2.internal] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[POST http://ip-10-0-60-152.ec2.internal:11000/oozie/v2/jobs?action=start&user=hadoop] error[E0307], E0307: Runtime error [App directory [null] does not exist and app definition cannot be created because of missing config value [oozie.jobs.api.generated.xml]] org.apache.oozie.servlet.XServletException: E0307: Runtime error [App directory [null] does not exist and app definition cannot be created because of missing config value [oozie.jobs.api.generated.xml]] at org.apache.oozie.servlet.V1JobsServlet.checkAndWriteApplicationXMLToHDFS(V1JobsServlet.java:175) at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:111) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:305) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772) at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592) at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:534) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) I checked again and again, and I have no idea what this "[App directory [null]" coming from. * I keep using the old XSD as workflow 0.3 and coordinator:0.5 under oozie 5.0, that should be supported right? * I didn't see any issue of the application path in the properties file, and as it works under "workflow" mode, so there should be no issue. * Since this error ONLY happens in the coordinator mode, and the only issue COULD come from the coordinator.xml. As I pasted above, it is so simple, and I really not sure where the issue comes from. * I even use the XSD from https://oozie.apache.org/docs/5.0.0/CoordinatorFunctionalSpec.html#Oozie_Coordinator_Schema_0.4 to validate my coordinator.xml, and it is valid. * So my questions are: * Where is this App directory [null] coming from? * The root reason for failure is due to AppDirectory is NULL, or definition cannot be created because of missing config value. If later, what kind of config value is missing we are talking about here? * If it is due to AppDirectory is NULL, what could cause it ONLY happens in Coord mode, but work fine in Workflow mode? Thanks Yong
