Re: Oozie 5.0 coordinator issue
Yes, you are right. The issue happened on EMR 5.22 for us, which comes with
Oozie 5.1.0
I will test a lower version of EMR with Oozie 5.0.0. From the JIRA, it looks
like this issue only happens since Oozie 5.1.0
For us, we are currently using EMR 5.12.2 with Oozie 4.3.0, and everything
works fine.
But we need to upgrade for newer Spark version.
Yong
From: Jerry Chabot
Sent: Monday, April 1, 2019 6:54 PM
To: user@oozie.apache.org
Subject: RE: Oozie 5.0 coordinator issue
What EMR version are you using? Can you confirm you are using Oozie 5.0? I ask
because I encountered a similar problem when we went from Oozie 5.0 to 5.1
while upgrading to EMR 5.22. I believe oozie.jobs.api.generated.xml was added
in Oozie 5.1. See https://issues.apache.org/jira/browse/OOZIE-3458,
https://issues.apache.org/jira/browse/OOZIE-3457 and related bugs for
details. An important point is the code reports "null" when that isn't the real
problem.
-Jerry
-Original Message-
From: Daniel Zhang [mailto:java8...@hotmail.com]
Sent: Monday, April 1, 2019 6:29 PM
To: user@oozie.apache.org
Subject: Oozie 5.0 coordinator issue
Hi, I am facing a strange issue on Oozie 5.0.
I have an Ozzie application that works fine in the workflow mode.
Here is the only 2 examples line of my oozie properties file:
oozie.wf.application.path=${appBaseDir}/workflow/workflow.xml
#oozie.coord.application.path=${appBaseDir}/workflow/coordinator.xml
And here is the content of the coordinator.xml file:
${appBaseDir}/workflow/workflow.xml
nameNode
${nameNode}
jobTracker
${jobTracker}
Now, this whole thing works on AWS EMR with Oozie 4.3. But with the new version
of AWS EMR comes with Oozie 5.0.0, I am facing the following strange error.
1) If I leave the properties file as above to running in the workflow mode by
"oozie job -run -config googleanalytics_aws_qa.properties file, everything
works under oozie 5.0
2) If I comment out the "wf" line above and uncomment out the "coord" line, I
got the following error:
[hadoop@ip-10-0-60-152 oozieJobs]$ oozie job -run -config
googleanalytics_aws_qa.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/lib/oozie/lib/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/lib/oozie/lib/slf4j-simple-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Error: E0307 : E0307: Runtime error [App directory [null] does not exist and
app definition cannot be created because of missing config value
[oozie.jobs.api.generated.xml]]
And here is the stack trace in the oozie.log:
2019-04-01 22:03:55,806 WARN V1JobsServlet:523 -
SERVER[ip-10-0-60-152.ec2.internal] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-]
ACTION[-] URL[POST
http://ip-10-0-60-152.ec2.internal:11000/oozie/v2/jobs?action=start=hadoop]
error[E0307], E0307: Runtime error [App directory [null] does not exist and
app definition cannot be created because of missing config value
[oozie.jobs.api.generated.xml]]
org.apache.oozie.servlet.XServletException: E0307: Runtime error [App directory
[null] does not exist and app definition cannot be created because of missing
config value [oozie.jobs.api.generated.xml]]
at
org.apache.oozie.servlet.V1JobsServlet.checkAndWriteApplicationXMLToHDFS(V1JobsServlet.java:175)
at
org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:111)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at
org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:305)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
org.eclipse.je
RE: Oozie 5.0 coordinator issue
What EMR version are you using? Can you confirm you are using Oozie 5.0? I ask
because I encountered a similar problem when we went from Oozie 5.0 to 5.1
while upgrading to EMR 5.22. I believe oozie.jobs.api.generated.xml was added
in Oozie 5.1. See https://issues.apache.org/jira/browse/OOZIE-3458,
https://issues.apache.org/jira/browse/OOZIE-3457 and related bugs for
details. An important point is the code reports "null" when that isn't the real
problem.
-Jerry
-Original Message-
From: Daniel Zhang [mailto:java8...@hotmail.com]
Sent: Monday, April 1, 2019 6:29 PM
To: user@oozie.apache.org
Subject: Oozie 5.0 coordinator issue
Hi, I am facing a strange issue on Oozie 5.0.
I have an Ozzie application that works fine in the workflow mode.
Here is the only 2 examples line of my oozie properties file:
oozie.wf.application.path=${appBaseDir}/workflow/workflow.xml
#oozie.coord.application.path=${appBaseDir}/workflow/coordinator.xml
And here is the content of the coordinator.xml file:
${appBaseDir}/workflow/workflow.xml
nameNode
${nameNode}
jobTracker
${jobTracker}
Now, this whole thing works on AWS EMR with Oozie 4.3. But with the new version
of AWS EMR comes with Oozie 5.0.0, I am facing the following strange error.
1) If I leave the properties file as above to running in the workflow mode by
"oozie job -run -config googleanalytics_aws_qa.properties file, everything
works under oozie 5.0
2) If I comment out the "wf" line above and uncomment out the "coord" line, I
got the following error:
[hadoop@ip-10-0-60-152 oozieJobs]$ oozie job -run -config
googleanalytics_aws_qa.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/lib/oozie/lib/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/lib/oozie/lib/slf4j-simple-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Error: E0307 : E0307: Runtime error [App directory [null] does not exist and
app definition cannot be created because of missing config value
[oozie.jobs.api.generated.xml]]
And here is the stack trace in the oozie.log:
2019-04-01 22:03:55,806 WARN V1JobsServlet:523 -
SERVER[ip-10-0-60-152.ec2.internal] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-]
ACTION[-] URL[POST
http://ip-10-0-60-152.ec2.internal:11000/oozie/v2/jobs?action=start=hadoop]
error[E0307], E0307: Runtime error [App directory [null] does not exist and
app definition cannot be created because of missing config value
[oozie.jobs.api.generated.xml]]
org.apache.oozie.servlet.XServletException: E0307: Runtime error [App directory
[null] does not exist and app definition cannot be created because of missing
config value [oozie.jobs.api.generated.xml]]
at
org.apache.oozie.servlet.V1JobsServlet.checkAndWriteApplicationXMLToHDFS(V1JobsServlet.java:175)
at
org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:111)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at
org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:305)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at
org.eclipse.jetty.servlet.ServletHand
Oozie 5.0 coordinator issue
Hi, I am facing a strange issue on Oozie 5.0.
I have an Ozzie application that works fine in the workflow mode.
Here is the only 2 examples line of my oozie properties file:
oozie.wf.application.path=${appBaseDir}/workflow/workflow.xml
#oozie.coord.application.path=${appBaseDir}/workflow/coordinator.xml
And here is the content of the coordinator.xml file:
${appBaseDir}/workflow/workflow.xml
nameNode
${nameNode}
jobTracker
${jobTracker}
Now, this whole thing works on AWS EMR with Oozie 4.3. But with the new version
of AWS EMR comes with Oozie 5.0.0, I am facing the following strange error.
1) If I leave the properties file as above to running in the workflow mode by
"oozie job -run -config googleanalytics_aws_qa.properties file, everything
works under oozie 5.0
2) If I comment out the "wf" line above and uncomment out the "coord" line, I
got the following error:
[hadoop@ip-10-0-60-152 oozieJobs]$ oozie job -run -config
googleanalytics_aws_qa.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/lib/oozie/lib/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/lib/oozie/lib/slf4j-simple-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Error: E0307 : E0307: Runtime error [App directory [null] does not exist and
app definition cannot be created because of missing config value
[oozie.jobs.api.generated.xml]]
And here is the stack trace in the oozie.log:
2019-04-01 22:03:55,806 WARN V1JobsServlet:523 -
SERVER[ip-10-0-60-152.ec2.internal] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-]
ACTION[-] URL[POST
http://ip-10-0-60-152.ec2.internal:11000/oozie/v2/jobs?action=start=hadoop]
error[E0307], E0307: Runtime error [App directory [null] does not exist and
app definition cannot be created because of missing config value
[oozie.jobs.api.generated.xml]]
org.apache.oozie.servlet.XServletException: E0307: Runtime error [App directory
[null] does not exist and app definition cannot be created because of missing
config value [oozie.jobs.api.generated.xml]]
at
org.apache.oozie.servlet.V1JobsServlet.checkAndWriteApplicationXMLToHDFS(V1JobsServlet.java:175)
at
org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:111)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at
org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:305)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
