[jira] [Created] (OOZIE-2928) Unit Tests using LocalOozie fail randomly
Denes Bodo created OOZIE-2928:
-
Summary: Unit Tests using LocalOozie fail randomly
Key: OOZIE-2928
URL: https://issues.apache.org/jira/browse/OOZIE-2928
Project: Oozie
Issue Type: Bug
Components: core
Affects Versions: 4.3.0
Reporter: Denes Bodo
Sometimes There is an erro message running unit tests: "LocalOozie is already
initialized" then fails all the tests using LocalOozie.
There is already a BindException that
{noformat}
java.net.BindException: Address already in use
at java.net.PlainSocketImpl.socketBind(Native Method)
at
java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376)
at java.net.ServerSocket.bind(ServerSocket.java:376)
at java.net.ServerSocket.(ServerSocket.java:237)
at
org.mortbay.jetty.bio.SocketConnector.newServerSocket(SocketConnector.java:80)
at org.mortbay.jetty.bio.SocketConnector.open(SocketConnector.java:73)
at
org.mortbay.jetty.AbstractConnector.doStart(AbstractConnector.java:283)
at
org.mortbay.jetty.bio.SocketConnector.doStart(SocketConnector.java:147)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.Server.doStart(Server.java:233)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at
org.apache.oozie.test.EmbeddedServletContainer.start(EmbeddedServletContainer.java:105)
at org.apache.oozie.local.LocalOozie.start(LocalOozie.java:81)
at
org.apache.oozie.command.coord.TestCoordRerunXCommand.setUp(TestCoordRerunXCommand.java:79)
at junit.framework.TestCase.runBare(TestCase.java:132)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:243)
at junit.framework.TestSuite.run(TestSuite.java:238)
at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:24)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
{noformat}
The source of problem seems to be in EmbeddedServletContainer.java. First we
create a socket, get its port number, release the socket and use this port to
create server:
{noformat}
ServerSocket ss = new ServerSocket(0);
port = ss.getLocalPort();
ss.close();
server.getConnectors()[0].setHost(host);
server.getConnectors()[0].setPort(port);
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OOZIE-2928) Unit Tests using LocalOozie fail randomly
[
https://issues.apache.org/jira/browse/OOZIE-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Bodo updated OOZIE-2928:
--
Attachment: 0001-Fix-random-LocalOozie-based-test-failure.patch
> Unit Tests using LocalOozie fail randomly
> -
>
> Key: OOZIE-2928
> URL: https://issues.apache.org/jira/browse/OOZIE-2928
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.0
>Reporter: Denes Bodo
> Labels: Servlet
> Attachments: 0001-Fix-random-LocalOozie-based-test-failure.patch
>
>
> Sometimes There is an erro message running unit tests: "LocalOozie is already
> initialized" then fails all the tests using LocalOozie.
> There is already a BindException that
> {noformat}
> type="java.net.BindException">java.net.BindException: Address already in use
> at java.net.PlainSocketImpl.socketBind(Native Method)
> at
> java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376)
> at java.net.ServerSocket.bind(ServerSocket.java:376)
> at java.net.ServerSocket.(ServerSocket.java:237)
> at
> org.mortbay.jetty.bio.SocketConnector.newServerSocket(SocketConnector.java:80)
> at org.mortbay.jetty.bio.SocketConnector.open(SocketConnector.java:73)
> at
> org.mortbay.jetty.AbstractConnector.doStart(AbstractConnector.java:283)
> at
> org.mortbay.jetty.bio.SocketConnector.doStart(SocketConnector.java:147)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> at org.mortbay.jetty.Server.doStart(Server.java:233)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> at
> org.apache.oozie.test.EmbeddedServletContainer.start(EmbeddedServletContainer.java:105)
> at org.apache.oozie.local.LocalOozie.start(LocalOozie.java:81)
> at
> org.apache.oozie.command.coord.TestCoordRerunXCommand.setUp(TestCoordRerunXCommand.java:79)
> at junit.framework.TestCase.runBare(TestCase.java:132)
> at junit.framework.TestResult$1.protect(TestResult.java:110)
> at junit.framework.TestResult.runProtected(TestResult.java:128)
> at junit.framework.TestResult.run(TestResult.java:113)
> at junit.framework.TestCase.run(TestCase.java:124)
> at junit.framework.TestSuite.runTest(TestSuite.java:243)
> at junit.framework.TestSuite.run(TestSuite.java:238)
> at
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:24)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
>
> {noformat}
> The source of problem seems to be in EmbeddedServletContainer.java. First we
> create a socket, get its port number, release the socket and use this port to
> create server:
> {noformat}
> ServerSocket ss = new ServerSocket(0);
> port = ss.getLocalPort();
> ss.close();
> server.getConnectors()[0].setHost(host);
> server.getConnectors()[0].setPort(port);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OOZIE-2928) Unit Tests using LocalOozie fail randomly
[
https://issues.apache.org/jira/browse/OOZIE-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Bodo updated OOZIE-2928:
--
Attachment: 0001-Fix-random-LocalOozie-based-test-failure.patch
Added parallel start test
> Unit Tests using LocalOozie fail randomly
> -
>
> Key: OOZIE-2928
> URL: https://issues.apache.org/jira/browse/OOZIE-2928
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.0
>Reporter: Denes Bodo
> Labels: Servlet
> Attachments: 0001-Fix-random-LocalOozie-based-test-failure.patch,
> 0001-Fix-random-LocalOozie-based-test-failure.patch
>
>
> Sometimes There is an erro message running unit tests: "LocalOozie is already
> initialized" then fails all the tests using LocalOozie.
> There is already a BindException that
> {noformat}
> type="java.net.BindException">java.net.BindException: Address already in use
> at java.net.PlainSocketImpl.socketBind(Native Method)
> at
> java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376)
> at java.net.ServerSocket.bind(ServerSocket.java:376)
> at java.net.ServerSocket.(ServerSocket.java:237)
> at
> org.mortbay.jetty.bio.SocketConnector.newServerSocket(SocketConnector.java:80)
> at org.mortbay.jetty.bio.SocketConnector.open(SocketConnector.java:73)
> at
> org.mortbay.jetty.AbstractConnector.doStart(AbstractConnector.java:283)
> at
> org.mortbay.jetty.bio.SocketConnector.doStart(SocketConnector.java:147)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> at org.mortbay.jetty.Server.doStart(Server.java:233)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> at
> org.apache.oozie.test.EmbeddedServletContainer.start(EmbeddedServletContainer.java:105)
> at org.apache.oozie.local.LocalOozie.start(LocalOozie.java:81)
> at
> org.apache.oozie.command.coord.TestCoordRerunXCommand.setUp(TestCoordRerunXCommand.java:79)
> at junit.framework.TestCase.runBare(TestCase.java:132)
> at junit.framework.TestResult$1.protect(TestResult.java:110)
> at junit.framework.TestResult.runProtected(TestResult.java:128)
> at junit.framework.TestResult.run(TestResult.java:113)
> at junit.framework.TestCase.run(TestCase.java:124)
> at junit.framework.TestSuite.runTest(TestSuite.java:243)
> at junit.framework.TestSuite.run(TestSuite.java:238)
> at
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:24)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
>
> {noformat}
> The source of problem seems to be in EmbeddedServletContainer.java. First we
> create a socket, get its port number, release the socket and use this port to
> create server:
> {noformat}
> ServerSocket ss = new ServerSocket(0);
> port = ss.getLocalPort();
> ss.close();
> server.getConnectors()[0].setHost(host);
> server.getConnectors()[0].setPort(port);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2928) Unit Tests using LocalOozie fail randomly
[
https://issues.apache.org/jira/browse/OOZIE-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16040885#comment-16040885
]
Denes Bodo commented on OOZIE-2928:
---
https://reviews.apache.org/r/59879/
> Unit Tests using LocalOozie fail randomly
> -
>
> Key: OOZIE-2928
> URL: https://issues.apache.org/jira/browse/OOZIE-2928
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.0
>Reporter: Denes Bodo
> Labels: Servlet
> Attachments: 0001-Fix-random-LocalOozie-based-test-failure.patch,
> 0001-Fix-random-LocalOozie-based-test-failure.patch
>
>
> Sometimes There is an erro message running unit tests: "LocalOozie is already
> initialized" then fails all the tests using LocalOozie.
> There is already a BindException that
> {noformat}
> type="java.net.BindException">java.net.BindException: Address already in use
> at java.net.PlainSocketImpl.socketBind(Native Method)
> at
> java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376)
> at java.net.ServerSocket.bind(ServerSocket.java:376)
> at java.net.ServerSocket.(ServerSocket.java:237)
> at
> org.mortbay.jetty.bio.SocketConnector.newServerSocket(SocketConnector.java:80)
> at org.mortbay.jetty.bio.SocketConnector.open(SocketConnector.java:73)
> at
> org.mortbay.jetty.AbstractConnector.doStart(AbstractConnector.java:283)
> at
> org.mortbay.jetty.bio.SocketConnector.doStart(SocketConnector.java:147)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> at org.mortbay.jetty.Server.doStart(Server.java:233)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> at
> org.apache.oozie.test.EmbeddedServletContainer.start(EmbeddedServletContainer.java:105)
> at org.apache.oozie.local.LocalOozie.start(LocalOozie.java:81)
> at
> org.apache.oozie.command.coord.TestCoordRerunXCommand.setUp(TestCoordRerunXCommand.java:79)
> at junit.framework.TestCase.runBare(TestCase.java:132)
> at junit.framework.TestResult$1.protect(TestResult.java:110)
> at junit.framework.TestResult.runProtected(TestResult.java:128)
> at junit.framework.TestResult.run(TestResult.java:113)
> at junit.framework.TestCase.run(TestCase.java:124)
> at junit.framework.TestSuite.runTest(TestSuite.java:243)
> at junit.framework.TestSuite.run(TestSuite.java:238)
> at
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:24)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
>
> {noformat}
> The source of problem seems to be in EmbeddedServletContainer.java. First we
> create a socket, get its port number, release the socket and use this port to
> create server:
> {noformat}
> ServerSocket ss = new ServerSocket(0);
> port = ss.getLocalPort();
> ss.close();
> server.getConnectors()[0].setHost(host);
> server.getConnectors()[0].setPort(port);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (OOZIE-2940) Possible NullPointerException
[
https://issues.apache.org/jira/browse/OOZIE-2940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Bodo reassigned OOZIE-2940:
-
Assignee: Denes Bodo
> Possible NullPointerException
> -
>
> Key: OOZIE-2940
> URL: https://issues.apache.org/jira/browse/OOZIE-2940
> Project: Oozie
> Issue Type: Bug
> Components: workflow
>Affects Versions: 4.3.0
>Reporter: AppChecker
>Assignee: Denes Bodo
> Labels: appchecker, static-analysis
>
> Hi.
> In [this
> code|https://github.com/apache/oozie/blob/eb168360c550943828d9fe2c7bfdcd4f5e830003/core/src/main/java/org/apache/oozie/WorkflowActionBean.java#L600]:
> {code:java}
> if (status == null || (status != Status.OK && status != Status.ERROR
> && status != Status.KILLED)) {
> throw new IllegalArgumentException("Action status must be OK,
> ERROR or KILLED. Received ["
> + status.toString() + "]");
> }
> {code}
> if status is null, status.toString() will be executed, so
> NullPointerException will be throw.
> This possible defect found by
> [AppChecker|https://npo-echelon.ru/en/solutions/appchecker.php].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-2940) Possible NullPointerException
[
https://issues.apache.org/jira/browse/OOZIE-2940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123222#comment-16123222
]
Denes Bodo commented on OOZIE-2940:
---
A possible fix can be found: https://reviews.apache.org/r/61216/
> Possible NullPointerException
> -
>
> Key: OOZIE-2940
> URL: https://issues.apache.org/jira/browse/OOZIE-2940
> Project: Oozie
> Issue Type: Bug
> Components: workflow
>Affects Versions: 4.3.0
>Reporter: AppChecker
>Assignee: Denes Bodo
> Labels: appchecker, static-analysis
>
> Hi.
> In [this
> code|https://github.com/apache/oozie/blob/eb168360c550943828d9fe2c7bfdcd4f5e830003/core/src/main/java/org/apache/oozie/WorkflowActionBean.java#L600]:
> {code:java}
> if (status == null || (status != Status.OK && status != Status.ERROR
> && status != Status.KILLED)) {
> throw new IllegalArgumentException("Action status must be OK,
> ERROR or KILLED. Received ["
> + status.toString() + "]");
> }
> {code}
> if status is null, status.toString() will be executed, so
> NullPointerException will be throw.
> This possible defect found by
> [AppChecker|https://npo-echelon.ru/en/solutions/appchecker.php].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-2847) Oozie Ha timing issue
[
https://issues.apache.org/jira/browse/OOZIE-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123403#comment-16123403
]
Denes Bodo commented on OOZIE-2847:
---
What do you think about https://reviews.apache.org/r/61208/ ?
> Oozie Ha timing issue
> -
>
> Key: OOZIE-2847
> URL: https://issues.apache.org/jira/browse/OOZIE-2847
> Project: Oozie
> Issue Type: Bug
> Components: HA
>Affects Versions: 4.3.0
>Reporter: Péter Gergő Barna
>Priority: Minor
> Attachments: OOZIE-2847.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Oozie Ha timing issue
> When Oozie is launching the mapper, it is writing a job id into a file on
> hdfs. Let's assume the ApplicationMaster is killed, and Oozie will make a
> second try, during recovery. On the second try, Oozie is trying to see if the
> previously written job id on hdfs matches the current job id. In most
> occasion, this will match. However, in the event when Oozie launcher is
> killed right in the middle when Oozie is in the process of writing id in the
> file, the Oozie file in hdfs is created, but the id has yet to be written to
> the file. During the next recovery, Oozie will mistakenly think the id exists
> in the file while the file is actually empty, therefore throwing this
> exception:
> {noformat}
> 2015-07-10
> 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|
> 2015-07-10 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|Console URL
> : http://dal-ha21:8088/proxy/application_1436507526035_0001/
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error Code
> : JA018
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error
> Message : Hadoop job Id mismatch, action file
> [hdfs://hdp2-ha2/user/hadoopqa/oozie-hado/003-150710041341636-oozie-hado-W/pig-node--pig/003-150710041341636-oozie-hado-W@pig-node@0]
> declares Id [null] current Id [job_1436507526035_0001]
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External ID
> : job_1436507526035_0001
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External
> Status : FAILED/KILLED
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Name
> : pig-node
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Retries
> : 0
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Tracker URI
> : dal-ha21:8032
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Type
> : pig
> 2015-07-10 05:56:58,158|beaver.machine|INFO|5208|1344|MainThread|Started
> : 2015-07-10 05:55:19 GMT
> 2015-07-10 05:56:58,160|beaver.machine|INFO|5208|1344|MainThread|Status
> : ERROR
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|Ended
> : 2015-07-10 05:56:42 GMT
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> Stats: null
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> ChildIDs : null
> 2015-07-10
> 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|
> Exception:
> 2015-07-10 05:56:18,658 INFO [main]
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file
> system [hdfs://hdp2-ha2:8020]
> 2015-07-10 05:56:18,665 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at
> hdfs://hdp2-ha2:8020/user/hadoopqa/.staging/job_1436507526035_0001/job_1436507526035_0001_1.jhist
> 2015-07-10 05:56:18,693 WARN [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Unable to parse prior job
> history, aborting recovery
> java.io.IOException: Incompatible event log version: null
> at
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:71)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1206)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1175)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1039)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1519)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.ap
[jira] [Assigned] (OOZIE-2847) Oozie Ha timing issue
[
https://issues.apache.org/jira/browse/OOZIE-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Bodo reassigned OOZIE-2847:
-
Assignee: Denes Bodo
> Oozie Ha timing issue
> -
>
> Key: OOZIE-2847
> URL: https://issues.apache.org/jira/browse/OOZIE-2847
> Project: Oozie
> Issue Type: Bug
> Components: HA
>Affects Versions: 4.3.0
>Reporter: Péter Gergő Barna
>Assignee: Denes Bodo
>Priority: Minor
> Attachments: OOZIE-2847.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Oozie Ha timing issue
> When Oozie is launching the mapper, it is writing a job id into a file on
> hdfs. Let's assume the ApplicationMaster is killed, and Oozie will make a
> second try, during recovery. On the second try, Oozie is trying to see if the
> previously written job id on hdfs matches the current job id. In most
> occasion, this will match. However, in the event when Oozie launcher is
> killed right in the middle when Oozie is in the process of writing id in the
> file, the Oozie file in hdfs is created, but the id has yet to be written to
> the file. During the next recovery, Oozie will mistakenly think the id exists
> in the file while the file is actually empty, therefore throwing this
> exception:
> {noformat}
> 2015-07-10
> 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|
> 2015-07-10 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|Console URL
> : http://dal-ha21:8088/proxy/application_1436507526035_0001/
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error Code
> : JA018
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error
> Message : Hadoop job Id mismatch, action file
> [hdfs://hdp2-ha2/user/hadoopqa/oozie-hado/003-150710041341636-oozie-hado-W/pig-node--pig/003-150710041341636-oozie-hado-W@pig-node@0]
> declares Id [null] current Id [job_1436507526035_0001]
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External ID
> : job_1436507526035_0001
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External
> Status : FAILED/KILLED
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Name
> : pig-node
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Retries
> : 0
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Tracker URI
> : dal-ha21:8032
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Type
> : pig
> 2015-07-10 05:56:58,158|beaver.machine|INFO|5208|1344|MainThread|Started
> : 2015-07-10 05:55:19 GMT
> 2015-07-10 05:56:58,160|beaver.machine|INFO|5208|1344|MainThread|Status
> : ERROR
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|Ended
> : 2015-07-10 05:56:42 GMT
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> Stats: null
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> ChildIDs : null
> 2015-07-10
> 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|
> Exception:
> 2015-07-10 05:56:18,658 INFO [main]
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file
> system [hdfs://hdp2-ha2:8020]
> 2015-07-10 05:56:18,665 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at
> hdfs://hdp2-ha2:8020/user/hadoopqa/.staging/job_1436507526035_0001/job_1436507526035_0001_1.jhist
> 2015-07-10 05:56:18,693 WARN [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Unable to parse prior job
> history, aborting recovery
> java.io.IOException: Incompatible event log version: null
> at
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:71)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1206)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1175)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1039)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1519)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
[jira] [Updated] (OOZIE-2940) Possible NullPointerException
[
https://issues.apache.org/jira/browse/OOZIE-2940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Bodo updated OOZIE-2940:
--
Attachment: 0001-OOZIE-2940-Possible-NullPointerException.patch
> Possible NullPointerException
> -
>
> Key: OOZIE-2940
> URL: https://issues.apache.org/jira/browse/OOZIE-2940
> Project: Oozie
> Issue Type: Bug
> Components: workflow
>Affects Versions: 4.3.0
>Reporter: AppChecker
>Assignee: Denes Bodo
> Labels: appchecker, static-analysis
> Attachments: 0001-OOZIE-2940-Possible-NullPointerException.patch
>
>
> Hi.
> In [this
> code|https://github.com/apache/oozie/blob/eb168360c550943828d9fe2c7bfdcd4f5e830003/core/src/main/java/org/apache/oozie/WorkflowActionBean.java#L600]:
> {code:java}
> if (status == null || (status != Status.OK && status != Status.ERROR
> && status != Status.KILLED)) {
> throw new IllegalArgumentException("Action status must be OK,
> ERROR or KILLED. Received ["
> + status.toString() + "]");
> }
> {code}
> if status is null, status.toString() will be executed, so
> NullPointerException will be throw.
> This possible defect found by
> [AppChecker|https://npo-echelon.ru/en/solutions/appchecker.php].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (OOZIE-3063) Sanitizing variables that are part of openjpa.ConnectionProperties
Denes Bodo created OOZIE-3063: - Summary: Sanitizing variables that are part of openjpa.ConnectionProperties Key: OOZIE-3063 URL: https://issues.apache.org/jira/browse/OOZIE-3063 Project: Oozie Issue Type: Bug Components: core Affects Versions: 4.2.0 Reporter: Denes Bodo There are values from oozie-site.xml which are not properly checked before they are passed into a comma-separated string property in JPAService class. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OOZIE-3063) Sanitizing variables that are part of openjpa.ConnectionProperties
[ https://issues.apache.org/jira/browse/OOZIE-3063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16174576#comment-16174576 ] Denes Bodo commented on OOZIE-3063: --- [~andras.piros] I'll provide code later today. Oozie should check for integer values and url to avoid accidentally written SQL statements in connection properties. > Sanitizing variables that are part of openjpa.ConnectionProperties > -- > > Key: OOZIE-3063 > URL: https://issues.apache.org/jira/browse/OOZIE-3063 > Project: Oozie > Issue Type: Bug > Components: core >Affects Versions: 4.2.0 >Reporter: Denes Bodo > > There are values from oozie-site.xml which are not properly checked before > they are passed into a comma-separated string property in JPAService class. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (OOZIE-3063) Sanitizing variables that are part of openjpa.ConnectionProperties
[ https://issues.apache.org/jira/browse/OOZIE-3063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Bodo reassigned OOZIE-3063: - Assignee: Denes Bodo > Sanitizing variables that are part of openjpa.ConnectionProperties > -- > > Key: OOZIE-3063 > URL: https://issues.apache.org/jira/browse/OOZIE-3063 > Project: Oozie > Issue Type: Bug > Components: core >Affects Versions: 4.2.0 >Reporter: Denes Bodo >Assignee: Denes Bodo > > There are values from oozie-site.xml which are not properly checked before > they are passed into a comma-separated string property in JPAService class. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-3063) Sanitizing variables that are part of openjpa.ConnectionProperties
[ https://issues.apache.org/jira/browse/OOZIE-3063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Bodo updated OOZIE-3063: -- Attachment: 0001-OOZIE-3063-Sanitizing-variables-that-are-part-of-ope.patch > Sanitizing variables that are part of openjpa.ConnectionProperties > -- > > Key: OOZIE-3063 > URL: https://issues.apache.org/jira/browse/OOZIE-3063 > Project: Oozie > Issue Type: Bug > Components: core >Affects Versions: 4.2.0 >Reporter: Denes Bodo >Assignee: Denes Bodo > Attachments: > 0001-OOZIE-3063-Sanitizing-variables-that-are-part-of-ope.patch > > > There are values from oozie-site.xml which are not properly checked before > they are passed into a comma-separated string property in JPAService class. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-3063) Sanitizing variables that are part of openjpa.ConnectionProperties
[ https://issues.apache.org/jira/browse/OOZIE-3063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Bodo updated OOZIE-3063: -- Attachment: (was: 0001-OOZIE-3063-Sanitizing-variables-that-are-part-of-ope.patch) > Sanitizing variables that are part of openjpa.ConnectionProperties > -- > > Key: OOZIE-3063 > URL: https://issues.apache.org/jira/browse/OOZIE-3063 > Project: Oozie > Issue Type: Bug > Components: core >Affects Versions: 4.2.0 >Reporter: Denes Bodo >Assignee: Denes Bodo > Attachments: > 0001-OOZIE-3063-Sanitizing-variables-that-are-part-of-ope.patch > > > There are values from oozie-site.xml which are not properly checked before > they are passed into a comma-separated string property in JPAService class. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-3063) Sanitizing variables that are part of openjpa.ConnectionProperties
[ https://issues.apache.org/jira/browse/OOZIE-3063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Bodo updated OOZIE-3063: -- Attachment: 0001-OOZIE-3063-Sanitizing-variables-that-are-part-of-ope.patch > Sanitizing variables that are part of openjpa.ConnectionProperties > -- > > Key: OOZIE-3063 > URL: https://issues.apache.org/jira/browse/OOZIE-3063 > Project: Oozie > Issue Type: Bug > Components: core >Affects Versions: 4.2.0 >Reporter: Denes Bodo >Assignee: Denes Bodo > Attachments: > 0001-OOZIE-3063-Sanitizing-variables-that-are-part-of-ope.patch > > > There are values from oozie-site.xml which are not properly checked before > they are passed into a comma-separated string property in JPAService class. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OOZIE-3063) Sanitizing variables that are part of openjpa.ConnectionProperties
[ https://issues.apache.org/jira/browse/OOZIE-3063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16176262#comment-16176262 ] Denes Bodo commented on OOZIE-3063: --- [~gezapeti] As I see the ConfigurationService class has static methods to read configuration. We can get int values, which is OK, but the methods can not throw ~"NumberFormatException", but fall back to default 0 value. Should I extend the getter to be able to sign problem with number format or should I use a separated checker method? I prefer the first option. Driver on classpath should be checked, I agree. > Sanitizing variables that are part of openjpa.ConnectionProperties > -- > > Key: OOZIE-3063 > URL: https://issues.apache.org/jira/browse/OOZIE-3063 > Project: Oozie > Issue Type: Bug > Components: core >Affects Versions: 4.2.0 >Reporter: Denes Bodo >Assignee: Denes Bodo > Attachments: > 0001-OOZIE-3063-Sanitizing-variables-that-are-part-of-ope.patch > > > There are values from oozie-site.xml which are not properly checked before > they are passed into a comma-separated string property in JPAService class. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-3063) Sanitizing variables that are part of openjpa.ConnectionProperties
[ https://issues.apache.org/jira/browse/OOZIE-3063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Bodo updated OOZIE-3063: -- Attachment: 0001-OOZIE-3063-Sanitizing-variables-that-are-part-of-ope.patch Second version. Added unit tests also. > Sanitizing variables that are part of openjpa.ConnectionProperties > -- > > Key: OOZIE-3063 > URL: https://issues.apache.org/jira/browse/OOZIE-3063 > Project: Oozie > Issue Type: Bug > Components: core >Affects Versions: 4.2.0 >Reporter: Denes Bodo >Assignee: Denes Bodo > Attachments: > 0001-OOZIE-3063-Sanitizing-variables-that-are-part-of-ope.patch, > 0001-OOZIE-3063-Sanitizing-variables-that-are-part-of-ope.patch > > > There are values from oozie-site.xml which are not properly checked before > they are passed into a comma-separated string property in JPAService class. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OOZIE-3067) Remove duplicated logic from ZKJobsConcurrencyService
Denes Bodo created OOZIE-3067: - Summary: Remove duplicated logic from ZKJobsConcurrencyService Key: OOZIE-3067 URL: https://issues.apache.org/jira/browse/OOZIE-3067 Project: Oozie Issue Type: Bug Components: core Affects Versions: 4.2.0 Reporter: Denes Bodo Assignee: Denes Bodo In ZKJobsConcurrencyService getServerUrls and getOtherServerUrls methods basically the same. However, the logic is written inside both methods. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-3067) Remove duplicated logic from ZKJobsConcurrencyService
[ https://issues.apache.org/jira/browse/OOZIE-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Bodo updated OOZIE-3067: -- Attachment: 0001-OOZIE-3067-Remove-duplicated-logic-from-ZKJobsConcur.patch My proposal. > Remove duplicated logic from ZKJobsConcurrencyService > - > > Key: OOZIE-3067 > URL: https://issues.apache.org/jira/browse/OOZIE-3067 > Project: Oozie > Issue Type: Bug > Components: core >Affects Versions: 4.2.0 >Reporter: Denes Bodo >Assignee: Denes Bodo > Attachments: > 0001-OOZIE-3067-Remove-duplicated-logic-from-ZKJobsConcur.patch > > > In ZKJobsConcurrencyService getServerUrls and getOtherServerUrls methods > basically the same. However, the logic is written inside both methods. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-3067) Remove duplicated logic from ZKJobsConcurrencyService
[ https://issues.apache.org/jira/browse/OOZIE-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Bodo updated OOZIE-3067: -- Attachment: OOZIE-3067.002.patch Fixed patch file. > Remove duplicated logic from ZKJobsConcurrencyService > - > > Key: OOZIE-3067 > URL: https://issues.apache.org/jira/browse/OOZIE-3067 > Project: Oozie > Issue Type: Bug > Components: core >Affects Versions: 4.2.0 >Reporter: Denes Bodo >Assignee: Denes Bodo > Attachments: > 0001-OOZIE-3067-Remove-duplicated-logic-from-ZKJobsConcur.patch, > OOZIE-3067.002.patch > > > In ZKJobsConcurrencyService getServerUrls and getOtherServerUrls methods > basically the same. However, the logic is written inside both methods. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-3063) Sanitizing variables that are part of openjpa.ConnectionProperties
[ https://issues.apache.org/jira/browse/OOZIE-3063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Bodo updated OOZIE-3063: -- Attachment: OOZIE-3063.003.patch Fixced patch file. I'll do the same in the future. Thanks [~gezapeti] > Sanitizing variables that are part of openjpa.ConnectionProperties > -- > > Key: OOZIE-3063 > URL: https://issues.apache.org/jira/browse/OOZIE-3063 > Project: Oozie > Issue Type: Bug > Components: core >Affects Versions: 4.2.0 >Reporter: Denes Bodo >Assignee: Denes Bodo > Attachments: > 0001-OOZIE-3063-Sanitizing-variables-that-are-part-of-ope.patch, > 0001-OOZIE-3063-Sanitizing-variables-that-are-part-of-ope.patch, > OOZIE-3063.003.patch > > > There are values from oozie-site.xml which are not properly checked before > they are passed into a comma-separated string property in JPAService class. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-3067) Remove duplicated logic from ZKJobsConcurrencyService
[ https://issues.apache.org/jira/browse/OOZIE-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Bodo updated OOZIE-3067: -- Priority: Minor (was: Major) > Remove duplicated logic from ZKJobsConcurrencyService > - > > Key: OOZIE-3067 > URL: https://issues.apache.org/jira/browse/OOZIE-3067 > Project: Oozie > Issue Type: Bug > Components: core >Affects Versions: 4.2.0 >Reporter: Denes Bodo >Assignee: Denes Bodo >Priority: Minor > Attachments: > 0001-OOZIE-3067-Remove-duplicated-logic-from-ZKJobsConcur.patch, > OOZIE-3067.002.patch > > > In ZKJobsConcurrencyService getServerUrls and getOtherServerUrls methods > basically the same. However, the logic is written inside both methods. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OOZIE-2871) when Enable Kerberos, Oozie perform tasks throw “Client cannot authenticate via:[TOKEN, KERBEROS]”
[ https://issues.apache.org/jira/browse/OOZIE-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16192814#comment-16192814 ] Denes Bodo commented on OOZIE-2871: --- [~yangfang] Were you able to solve this issue? > when Enable Kerberos, Oozie perform tasks throw “Client cannot authenticate > via:[TOKEN, KERBEROS]” > --- > > Key: OOZIE-2871 > URL: https://issues.apache.org/jira/browse/OOZIE-2871 > Project: Oozie > Issue Type: Bug > Components: security >Affects Versions: 4.2.0 > Environment: Oozie version :4.2.0 > Hadoop version:2.7.2 > Both Oozie and Hadoop are enabled kerberos. >Reporter: yangfang >Priority: Critical > > When Oozie and Hadoop both enabled kerberos， I submitted a mapreduce job > to oozie，then I got the error as below： > 2017-04-27 13:37:12,677 WARN MapReduceActionExecutor: 523 - SERVER[zdh143] > USER[mr] GROUP[-] TOKEN[] APP[map-reduce-wf] > JOB[008-170427133546167-oozie-mr-W] > ACTION[008-170427133546167-oozie-mr-W@mr-node] Launcher exception: Failed > on local exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[TOKEN, KERBEROS]; Host Details : local host is: "zdh142/10.43.183.142"; > destination host is: "zdh143":9000; > java.io.IOException: Failed on local exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[TOKEN, KERBEROS]; Host Details : local host is: "zdh142/10.43.183.142"; > destination host is: "zdh143":9000; > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:773) > at org.apache.hadoop.ipc.Client.call(Client.java:1479) > at org.apache.hadoop.ipc.Client.call(Client.java:1412) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) > at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy18.getFileInfo(Unknown Source) > at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108) > at > org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305) > at > org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426) > at > org.apache.hadoop.mapred.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:130) > at > org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:268) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1299) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1296) > at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575) > at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) > at > org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:570) > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:561) > at > org.apache.oozie.action.hadoop.MapReduceMain.submitJob(MapReduceMain.java:102) > at > org.apache.oozie.action.hadoop.MapReduceMain.run(MapReduceMain.java:64) > at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47) > at > org.apache.oozie.action.hadoop.MapReduceMain.main(MapReduceMain.java:38) > at sun.reflect.NativeMethodAccessorIm
[jira] [Commented] (OOZIE-2871) when Enable Kerberos, Oozie perform tasks throw “Client cannot authenticate via:[TOKEN, KERBEROS]”
[ https://issues.apache.org/jira/browse/OOZIE-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16196807#comment-16196807 ] Denes Bodo commented on OOZIE-2871: --- [~gezapeti] I think yes, based on "Cannot do cross cluster distcp" between two secure clusters. > when Enable Kerberos, Oozie perform tasks throw “Client cannot authenticate > via:[TOKEN, KERBEROS]” > --- > > Key: OOZIE-2871 > URL: https://issues.apache.org/jira/browse/OOZIE-2871 > Project: Oozie > Issue Type: Bug > Components: security >Affects Versions: 4.2.0 > Environment: Oozie version :4.2.0 > Hadoop version:2.7.2 > Both Oozie and Hadoop are enabled kerberos. >Reporter: yangfang >Priority: Critical > > When Oozie and Hadoop both enabled kerberos， I submitted a mapreduce job > to oozie，then I got the error as below： > 2017-04-27 13:37:12,677 WARN MapReduceActionExecutor: 523 - SERVER[zdh143] > USER[mr] GROUP[-] TOKEN[] APP[map-reduce-wf] > JOB[008-170427133546167-oozie-mr-W] > ACTION[008-170427133546167-oozie-mr-W@mr-node] Launcher exception: Failed > on local exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[TOKEN, KERBEROS]; Host Details : local host is: "zdh142/10.43.183.142"; > destination host is: "zdh143":9000; > java.io.IOException: Failed on local exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[TOKEN, KERBEROS]; Host Details : local host is: "zdh142/10.43.183.142"; > destination host is: "zdh143":9000; > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:773) > at org.apache.hadoop.ipc.Client.call(Client.java:1479) > at org.apache.hadoop.ipc.Client.call(Client.java:1412) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) > at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy18.getFileInfo(Unknown Source) > at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108) > at > org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305) > at > org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426) > at > org.apache.hadoop.mapred.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:130) > at > org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:268) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1299) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1296) > at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575) > at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) > at > org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:570) > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:561) > at > org.apache.oozie.action.hadoop.MapReduceMain.submitJob(MapReduceMain.java:102) > at > org.apache.oozie.action.hadoop.MapReduceMain.run(MapReduceMain.java:64) > at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47) > at > org.apache.oozie.action.hadoop.MapReduceMain.main(MapReduceMain.java
[jira] [Comment Edited] (OOZIE-2871) when Enable Kerberos, Oozie perform tasks throw “Client cannot authenticate via:[TOKEN, KERBEROS]”
[ https://issues.apache.org/jira/browse/OOZIE-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16196807#comment-16196807 ] Denes Bodo edited comment on OOZIE-2871 at 10/9/17 11:00 AM: - [~gezapeti] I think yes, based on "Cannot do cross cluster distcp" between two secure clusters. Oozie does not pass any HDFS delegation token to the distcp job. was (Author: dionusos): [~gezapeti] I think yes, based on "Cannot do cross cluster distcp" between two secure clusters. > when Enable Kerberos, Oozie perform tasks throw “Client cannot authenticate > via:[TOKEN, KERBEROS]” > --- > > Key: OOZIE-2871 > URL: https://issues.apache.org/jira/browse/OOZIE-2871 > Project: Oozie > Issue Type: Bug > Components: security >Affects Versions: 4.2.0 > Environment: Oozie version :4.2.0 > Hadoop version:2.7.2 > Both Oozie and Hadoop are enabled kerberos. >Reporter: yangfang >Priority: Critical > > When Oozie and Hadoop both enabled kerberos， I submitted a mapreduce job > to oozie，then I got the error as below： > 2017-04-27 13:37:12,677 WARN MapReduceActionExecutor: 523 - SERVER[zdh143] > USER[mr] GROUP[-] TOKEN[] APP[map-reduce-wf] > JOB[008-170427133546167-oozie-mr-W] > ACTION[008-170427133546167-oozie-mr-W@mr-node] Launcher exception: Failed > on local exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[TOKEN, KERBEROS]; Host Details : local host is: "zdh142/10.43.183.142"; > destination host is: "zdh143":9000; > java.io.IOException: Failed on local exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[TOKEN, KERBEROS]; Host Details : local host is: "zdh142/10.43.183.142"; > destination host is: "zdh143":9000; > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:773) > at org.apache.hadoop.ipc.Client.call(Client.java:1479) > at org.apache.hadoop.ipc.Client.call(Client.java:1412) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) > at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy18.getFileInfo(Unknown Source) > at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108) > at > org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305) > at > org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426) > at > org.apache.hadoop.mapred.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:130) > at > org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:268) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1299) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1296) > at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575) > at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) > at > org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:570) > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:561) > at > org.apache.oozie.action.hadoop.MapReduceMain.submitJob(MapReduceMain.java:102) > a
[jira] [Commented] (OOZIE-2871) when Enable Kerberos, Oozie perform tasks throw “Client cannot authenticate via:[TOKEN, KERBEROS]”
[ https://issues.apache.org/jira/browse/OOZIE-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16210735#comment-16210735 ] Denes Bodo commented on OOZIE-2871: --- [~asasvari] A very similar issue can be reproducible between two secure clusters with the same realm. In that case using hadoop distcp command from command line works as expected, but when we ask Oozie it fails with the above error (and stack trace). Oozie version is 4.2. > when Enable Kerberos, Oozie perform tasks throw “Client cannot authenticate > via:[TOKEN, KERBEROS]” > --- > > Key: OOZIE-2871 > URL: https://issues.apache.org/jira/browse/OOZIE-2871 > Project: Oozie > Issue Type: Bug > Components: security >Affects Versions: 4.2.0 > Environment: Oozie version :4.2.0 > Hadoop version:2.7.2 > Both Oozie and Hadoop are enabled kerberos. >Reporter: yangfang >Priority: Critical > > When Oozie and Hadoop both enabled kerberos， I submitted a mapreduce job > to oozie，then I got the error as below： > 2017-04-27 13:37:12,677 WARN MapReduceActionExecutor: 523 - SERVER[zdh143] > USER[mr] GROUP[-] TOKEN[] APP[map-reduce-wf] > JOB[008-170427133546167-oozie-mr-W] > ACTION[008-170427133546167-oozie-mr-W@mr-node] Launcher exception: Failed > on local exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[TOKEN, KERBEROS]; Host Details : local host is: "zdh142/10.43.183.142"; > destination host is: "zdh143":9000; > java.io.IOException: Failed on local exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[TOKEN, KERBEROS]; Host Details : local host is: "zdh142/10.43.183.142"; > destination host is: "zdh143":9000; > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:773) > at org.apache.hadoop.ipc.Client.call(Client.java:1479) > at org.apache.hadoop.ipc.Client.call(Client.java:1412) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) > at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy18.getFileInfo(Unknown Source) > at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108) > at > org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305) > at > org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426) > at > org.apache.hadoop.mapred.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:130) > at > org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:268) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1299) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1296) > at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575) > at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) > at > org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:570) > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:561) > at > org.apache.oozie.action.hadoop.MapReduceMain.submitJob(MapReduceMain.java:102) > at > org.apache.oozie.action.hadoop.MapReduceMain.run(MapRedu
[jira] [Updated] (OOZIE-2871) when Enable Kerberos, Oozie perform tasks throw “Client cannot authenticate via:[TOKEN, KERBEROS]”
[ https://issues.apache.org/jira/browse/OOZIE-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Bodo updated OOZIE-2871: -- Attachment: secure_multicluster_distcp_workflow.xml I've uploaded a workflow.xml file which can be used to reproduce the issue. Node and cluster names are masked. In job.properties we set 'oozie.use.system.libpath=true' > when Enable Kerberos, Oozie perform tasks throw “Client cannot authenticate > via:[TOKEN, KERBEROS]” > --- > > Key: OOZIE-2871 > URL: https://issues.apache.org/jira/browse/OOZIE-2871 > Project: Oozie > Issue Type: Bug > Components: security >Affects Versions: 4.2.0 > Environment: Oozie version :4.2.0 > Hadoop version:2.7.2 > Both Oozie and Hadoop are enabled kerberos. >Reporter: yangfang >Priority: Critical > Attachments: secure_multicluster_distcp_workflow.xml > > > When Oozie and Hadoop both enabled kerberos， I submitted a mapreduce job > to oozie，then I got the error as below： > 2017-04-27 13:37:12,677 WARN MapReduceActionExecutor: 523 - SERVER[zdh143] > USER[mr] GROUP[-] TOKEN[] APP[map-reduce-wf] > JOB[008-170427133546167-oozie-mr-W] > ACTION[008-170427133546167-oozie-mr-W@mr-node] Launcher exception: Failed > on local exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[TOKEN, KERBEROS]; Host Details : local host is: "zdh142/10.43.183.142"; > destination host is: "zdh143":9000; > java.io.IOException: Failed on local exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[TOKEN, KERBEROS]; Host Details : local host is: "zdh142/10.43.183.142"; > destination host is: "zdh143":9000; > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:773) > at org.apache.hadoop.ipc.Client.call(Client.java:1479) > at org.apache.hadoop.ipc.Client.call(Client.java:1412) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) > at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy18.getFileInfo(Unknown Source) > at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108) > at > org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305) > at > org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426) > at > org.apache.hadoop.mapred.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:130) > at > org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:268) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1299) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1296) > at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575) > at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) > at > org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:570) > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:561) > at > org.apache.oozie.action.hadoop.MapReduceMain.submitJob(MapReduceMain.java:102) > at > org.apache.oozie.action.hadoop.MapReduceMain.run(MapReduceMain.java:64) > at org.apache.oozi
[jira] [Updated] (OOZIE-2871) when Enable Kerberos, Oozie perform tasks throw “Client cannot authenticate via:[TOKEN, KERBEROS]”
[ https://issues.apache.org/jira/browse/OOZIE-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Bodo updated OOZIE-2871: -- Attachment: OOZIE-2871.patch > when Enable Kerberos, Oozie perform tasks throw “Client cannot authenticate > via:[TOKEN, KERBEROS]” > --- > > Key: OOZIE-2871 > URL: https://issues.apache.org/jira/browse/OOZIE-2871 > Project: Oozie > Issue Type: Bug > Components: security >Affects Versions: 4.2.0 > Environment: Oozie version :4.2.0 > Hadoop version:2.7.2 > Both Oozie and Hadoop are enabled kerberos. >Reporter: yangfang >Priority: Critical > Attachments: OOZIE-2871.patch, secure_multicluster_distcp_workflow.xml > > > When Oozie and Hadoop both enabled kerberos， I submitted a mapreduce job > to oozie，then I got the error as below： > 2017-04-27 13:37:12,677 WARN MapReduceActionExecutor: 523 - SERVER[zdh143] > USER[mr] GROUP[-] TOKEN[] APP[map-reduce-wf] > JOB[008-170427133546167-oozie-mr-W] > ACTION[008-170427133546167-oozie-mr-W@mr-node] Launcher exception: Failed > on local exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[TOKEN, KERBEROS]; Host Details : local host is: "zdh142/10.43.183.142"; > destination host is: "zdh143":9000; > java.io.IOException: Failed on local exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[TOKEN, KERBEROS]; Host Details : local host is: "zdh142/10.43.183.142"; > destination host is: "zdh143":9000; > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:773) > at org.apache.hadoop.ipc.Client.call(Client.java:1479) > at org.apache.hadoop.ipc.Client.call(Client.java:1412) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) > at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy18.getFileInfo(Unknown Source) > at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108) > at > org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305) > at > org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426) > at > org.apache.hadoop.mapred.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:130) > at > org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:268) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1299) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1296) > at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575) > at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) > at > org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:570) > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:561) > at > org.apache.oozie.action.hadoop.MapReduceMain.submitJob(MapReduceMain.java:102) > at > org.apache.oozie.action.hadoop.MapReduceMain.run(MapReduceMain.java:64) > at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47) > at > org.apache.oozie.action.hadoop.MapReduceMain.main(MapReduceMain.java:38) > at sun.reflect.NativeMetho
[jira] [Commented] (OOZIE-2985) If LauncherAM fails, Oozie is not notified in a timely manner
[
https://issues.apache.org/jira/browse/OOZIE-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16218630#comment-16218630
]
Denes Bodo commented on OOZIE-2985:
---
I have experienced the same issue when tried to run the example distcp acrtion.
Both the ClassNotFoundException for LauncherAM and the long wait.
> If LauncherAM fails, Oozie is not notified in a timely manner
> -
>
> Key: OOZIE-2985
> URL: https://issues.apache.org/jira/browse/OOZIE-2985
> Project: Oozie
> Issue Type: Bug
>Reporter: Attila Sasvari
>
> I've noticed if LauncherAM fails, Oozie is notified about the launcher's
> failure with a lot of delay. It gives the impression that the workflow is
> running.
> {{oozie job -oozie http://localhost:11000/oozie -config
> examples/apps/datelist-java-main/job.properties -info
> 000-170712153835057-oozie-asas-W}}
> {code}
> 000-170712153835057-oozie-asas-W@java1
> RUNNING application_1499866588585_0001RUNNING-
> {code}
> I've looked at yarn logs for the launcher and seen that the launcher failed.
> For example, in my case , during development, oozie-sharelib launcher was not
> found:
> {code}
> Error: Could not find or load main class
> org.apache.oozie.action.hadoop.LauncherAM
> {code}
> The problem is only after the specified timeout (by default 10 minutes) we
> see that the workflow has actually failed /errored.
> {code}
> Created : 2017-07-12 13:38 GMT
> Started : 2017-07-12 13:38 GMT
> Last Modified : 2017-07-12 13:49 GMT
> ...
> 000-170712153835057-oozie-asas-W@java1
> ERROR application_1499866588585_0001FAILED/KILLED-
> {code}
> The problem might be that in {{JavaActionExecutor}} in the {{start()}} method
> the check is too fast.
> {code}
> LOG.debug("Starting action " + action.getId() + " getting Action File
> System");
> FileSystem actionFs = context.getAppFileSystem();
> LOG.debug("Preparing action Dir through copying " +
> context.getActionDir());
> prepareActionDir(actionFs, context);
> LOG.debug("Action Dir is ready. Submitting the action ");
> submitLauncher(actionFs, context, action);
> LOG.debug("Action submit completed. Performing check ");
> check(context, action);
> LOG.debug("Action check is done after submission
> {code}
> There should be some delay after {{submitLauncher()}} before {{check()}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-2985) If LauncherAM fails, Oozie is not notified in a timely manner
[
https://issues.apache.org/jira/browse/OOZIE-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16218807#comment-16218807
]
Denes Bodo commented on OOZIE-2985:
---
[~pbacsko] Am I right that Oozie can provide a notification URL to the job that
can call back to Oozie? Can we use the same method in LauncherAM to notify when
the main() starts running?
> If LauncherAM fails, Oozie is not notified in a timely manner
> -
>
> Key: OOZIE-2985
> URL: https://issues.apache.org/jira/browse/OOZIE-2985
> Project: Oozie
> Issue Type: Bug
>Reporter: Attila Sasvari
>
> I've noticed if LauncherAM fails, Oozie is notified about the launcher's
> failure with a lot of delay. It gives the impression that the workflow is
> running.
> {{oozie job -oozie http://localhost:11000/oozie -config
> examples/apps/datelist-java-main/job.properties -info
> 000-170712153835057-oozie-asas-W}}
> {code}
> 000-170712153835057-oozie-asas-W@java1
> RUNNING application_1499866588585_0001RUNNING-
> {code}
> I've looked at yarn logs for the launcher and seen that the launcher failed.
> For example, in my case , during development, oozie-sharelib launcher was not
> found:
> {code}
> Error: Could not find or load main class
> org.apache.oozie.action.hadoop.LauncherAM
> {code}
> The problem is only after the specified timeout (by default 10 minutes) we
> see that the workflow has actually failed /errored.
> {code}
> Created : 2017-07-12 13:38 GMT
> Started : 2017-07-12 13:38 GMT
> Last Modified : 2017-07-12 13:49 GMT
> ...
> 000-170712153835057-oozie-asas-W@java1
> ERROR application_1499866588585_0001FAILED/KILLED-
> {code}
> The problem might be that in {{JavaActionExecutor}} in the {{start()}} method
> the check is too fast.
> {code}
> LOG.debug("Starting action " + action.getId() + " getting Action File
> System");
> FileSystem actionFs = context.getAppFileSystem();
> LOG.debug("Preparing action Dir through copying " +
> context.getActionDir());
> prepareActionDir(actionFs, context);
> LOG.debug("Action Dir is ready. Submitting the action ");
> submitLauncher(actionFs, context, action);
> LOG.debug("Action submit completed. Performing check ");
> check(context, action);
> LOG.debug("Action check is done after submission
> {code}
> There should be some delay after {{submitLauncher()}} before {{check()}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-3091) Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: org/apache/avro/mapred/AvroWrapper"
[
https://issues.apache.org/jira/browse/OOZIE-3091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16225681#comment-16225681
]
Denes Bodo commented on OOZIE-3091:
---
[~Prabhu Joseph] I could help you uploading patch file. Please let me know if I
can help you.
> Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError:
> org/apache/avro/mapred/AvroWrapper"
> ---
>
> Key: OOZIE-3091
> URL: https://issues.apache.org/jira/browse/OOZIE-3091
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: 4.2.0
>Reporter: Prabhu Joseph
>
> Oozie Sqoop Action which does Import as avro fails with below.
> avro-mapred-1.8.0-hadoop2.jar need to be included in Oozie Sqoop Sharelib
> {code}
> 2017-10-19 09:45:25,349 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.lang.RuntimeException:
> java.lang.reflect.InvocationTargetException
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:134)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:745)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:132)
> ... 7 more
> Caused by: java.lang.NoClassDefFoundError: org/apache/avro/mapred/AvroWrapper
> at
> org.apache.sqoop.mapreduce.AvroImportMapper.(AvroImportMapper.java:43)
> ... 12 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.avro.mapred.AvroWrapper
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 13 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-3091) Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: org/apache/avro/mapred/AvroWrapper"
[
https://issues.apache.org/jira/browse/OOZIE-3091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226329#comment-16226329
]
Denes Bodo commented on OOZIE-3091:
---
[~Prabhu Joseph] I cannot find you in the list.
[~andras.piros] Can you please help us? Thanks.
> Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError:
> org/apache/avro/mapred/AvroWrapper"
> ---
>
> Key: OOZIE-3091
> URL: https://issues.apache.org/jira/browse/OOZIE-3091
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: 4.2.0
>Reporter: Prabhu Joseph
>
> Oozie Sqoop Action which does Import as avro fails with below.
> avro-mapred-1.8.0-hadoop2.jar need to be included in Oozie Sqoop Sharelib
> {code}
> 2017-10-19 09:45:25,349 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.lang.RuntimeException:
> java.lang.reflect.InvocationTargetException
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:134)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:745)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:132)
> ... 7 more
> Caused by: java.lang.NoClassDefFoundError: org/apache/avro/mapred/AvroWrapper
> at
> org.apache.sqoop.mapreduce.AvroImportMapper.(AvroImportMapper.java:43)
> ... 12 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.avro.mapred.AvroWrapper
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 13 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (OOZIE-3109) Good coding practice
Denes Bodo created OOZIE-3109: - Summary: Good coding practice Key: OOZIE-3109 URL: https://issues.apache.org/jira/browse/OOZIE-3109 Project: Oozie Issue Type: Bug Affects Versions: 4.3.0 Reporter: Denes Bodo Assignee: Denes Bodo That can be a security problem if we do not encode the stream read from log files before transferred to the web browser. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-3109) Good coding practice
[ https://issues.apache.org/jira/browse/OOZIE-3109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Bodo updated OOZIE-3109: -- Attachment: OOZIE-3109-v001.patch > Good coding practice > > > Key: OOZIE-3109 > URL: https://issues.apache.org/jira/browse/OOZIE-3109 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Denes Bodo >Assignee: Denes Bodo > Attachments: OOZIE-3109-v001.patch > > > That can be a security problem if we do not encode the stream read from log > files before transferred to the web browser. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-3109) Escape log-streaming's HTML-specific characters
[ https://issues.apache.org/jira/browse/OOZIE-3109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Bodo updated OOZIE-3109: -- Summary: Escape log-streaming's HTML-specific characters (was: Good coding practice) > Escape log-streaming's HTML-specific characters > --- > > Key: OOZIE-3109 > URL: https://issues.apache.org/jira/browse/OOZIE-3109 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Denes Bodo >Assignee: Denes Bodo > Attachments: OOZIE-3109-v001.patch > > > That can be a security problem if we do not encode the stream read from log > files before transferred to the web browser. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-3138) TestSparkMain.testMain UT fails
[
https://issues.apache.org/jira/browse/OOZIE-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Bodo updated OOZIE-3138:
--
Component/s: tests
> TestSparkMain.testMain UT fails
> ---
>
> Key: OOZIE-3138
> URL: https://issues.apache.org/jira/browse/OOZIE-3138
> Project: Oozie
> Issue Type: Test
> Components: tests
>Affects Versions: 4.2.0
>Reporter: Yesha Vora
>
> TestSparkMain.testMain UT fails with below error
> {code}
> Error Message
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient
> Stacktrace
> com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2263)
> at com.google.common.cache.LocalCache.get(LocalCache.java:4000)
> at
> com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4789)
> at
> org.apache.hive.hcatalog.common.HiveClientCache.getOrCreate(HiveClientCache.java:291)
> at
> org.apache.hive.hcatalog.common.HiveClientCache.get(HiveClientCache.java:273)
> at
> org.apache.hive.hcatalog.common.HCatUtil.getHiveMetastoreClient(HCatUtil.java:558)
> at
> org.apache.hive.hcatalog.api.HCatClientHMSImpl.initialize(HCatClientHMSImpl.java:823)
> at org.apache.hive.hcatalog.api.HCatClient.create(HCatClient.java:71)
> at org.apache.oozie.test.MiniHCatServer.start(MiniHCatServer.java:87)
> at
> org.apache.oozie.test.XTestCase.setupHCatalogServer(XTestCase.java:998)
> at org.apache.oozie.test.XTestCase.setUp(XTestCase.java:405)
> at org.apache.oozie.test.XTestCase.setUp(XTestCase.java:281)
> at org.apache.oozie.test.XFsTestCase.setUp(XFsTestCase.java:61)
> at junit.framework.TestCase.runBare(TestCase.java:132)
> at junit.framework.TestResult$1.protect(TestResult.java:110)
> at junit.framework.TestResult.runProtected(TestResult.java:128)
> at junit.framework.TestResult.run(TestResult.java:113)
> at junit.framework.TestCase.run(TestCase.java:124)
> at junit.framework.TestSuite.runTest(TestSuite.java:243)
> at junit.framework.TestSuite.run(TestSuite.java:238)
> at
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:24)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Unable to instantiate
> org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1566)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:92)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:138)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:124)
> at
> org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:295)
> at
> org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:291)
> at
> com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4792)
> at
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
> at
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
> at
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)
> ... 28 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1564)
> ... 38 more
> Caused by: MetaException
[jira] [Updated] (OOZIE-3138) TestSparkMain.testMain UT fails
[
https://issues.apache.org/jira/browse/OOZIE-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Bodo updated OOZIE-3138:
--
Affects Version/s: 4.2.0
> TestSparkMain.testMain UT fails
> ---
>
> Key: OOZIE-3138
> URL: https://issues.apache.org/jira/browse/OOZIE-3138
> Project: Oozie
> Issue Type: Test
> Components: tests
>Affects Versions: 4.2.0
>Reporter: Yesha Vora
>
> TestSparkMain.testMain UT fails with below error
> {code}
> Error Message
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient
> Stacktrace
> com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2263)
> at com.google.common.cache.LocalCache.get(LocalCache.java:4000)
> at
> com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4789)
> at
> org.apache.hive.hcatalog.common.HiveClientCache.getOrCreate(HiveClientCache.java:291)
> at
> org.apache.hive.hcatalog.common.HiveClientCache.get(HiveClientCache.java:273)
> at
> org.apache.hive.hcatalog.common.HCatUtil.getHiveMetastoreClient(HCatUtil.java:558)
> at
> org.apache.hive.hcatalog.api.HCatClientHMSImpl.initialize(HCatClientHMSImpl.java:823)
> at org.apache.hive.hcatalog.api.HCatClient.create(HCatClient.java:71)
> at org.apache.oozie.test.MiniHCatServer.start(MiniHCatServer.java:87)
> at
> org.apache.oozie.test.XTestCase.setupHCatalogServer(XTestCase.java:998)
> at org.apache.oozie.test.XTestCase.setUp(XTestCase.java:405)
> at org.apache.oozie.test.XTestCase.setUp(XTestCase.java:281)
> at org.apache.oozie.test.XFsTestCase.setUp(XFsTestCase.java:61)
> at junit.framework.TestCase.runBare(TestCase.java:132)
> at junit.framework.TestResult$1.protect(TestResult.java:110)
> at junit.framework.TestResult.runProtected(TestResult.java:128)
> at junit.framework.TestResult.run(TestResult.java:113)
> at junit.framework.TestCase.run(TestCase.java:124)
> at junit.framework.TestSuite.runTest(TestSuite.java:243)
> at junit.framework.TestSuite.run(TestSuite.java:238)
> at
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:24)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Unable to instantiate
> org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1566)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:92)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:138)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:124)
> at
> org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:295)
> at
> org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:291)
> at
> com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4792)
> at
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
> at
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
> at
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)
> ... 28 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1564)
> ... 38 more
> Caused by: MetaExc
[jira] [Created] (OOZIE-3139) Oozie validates workflow incorrectly
Denes Bodo created OOZIE-3139:
-
Summary: Oozie validates workflow incorrectly
Key: OOZIE-3139
URL: https://issues.apache.org/jira/browse/OOZIE-3139
Project: Oozie
Issue Type: Bug
Components: core
Affects Versions: 4.3.0
Reporter: Denes Bodo
Assignee: Denes Bodo
Priority: Critical
We found that after OOZIE-1978 the attached workflow becomes invalid. Aa we
found out the following line brings the problem:
https://github.com/apache/oozie/commit/8e9b9042b3270dc5ff975c44a5c977fcc41250e4#diff-a38ae78886b7c3afb711d32cfb4dbbbdR267
The following workflow is problematic:
{noformat}
${jobTracker}
${nameNode}
${myscript}
${myscriptPath}
${jobTracker}
${nameNode}
${myscript}
${myscriptPath}
${jobTracker}
${nameNode}
${myscript}
${myscriptPath}
${jobTracker}
${nameNode}
${myscript}
${myscriptPath}
${jobTracker}
${nameNode}
${myscript}
${myscriptPath}
Shell action failed, error
message[${wf:errorMessage(wf:lastErrorNode())}]
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-3139) Oozie validates workflow incorrectly
[
https://issues.apache.org/jira/browse/OOZIE-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Bodo updated OOZIE-3139:
--
Attachment: workflow.pdf
> Oozie validates workflow incorrectly
>
>
> Key: OOZIE-3139
> URL: https://issues.apache.org/jira/browse/OOZIE-3139
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Attachments: workflow.pdf
>
>
> We found that after OOZIE-1978 the attached workflow becomes invalid. Aa we
> found out the following line brings the problem:
> https://github.com/apache/oozie/commit/8e9b9042b3270dc5ff975c44a5c977fcc41250e4#diff-a38ae78886b7c3afb711d32cfb4dbbbdR267
> The following workflow is problematic:
> {noformat}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> Shell action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-3139) Oozie validates workflow incorrectly
[
https://issues.apache.org/jira/browse/OOZIE-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16278695#comment-16278695
]
Denes Bodo commented on OOZIE-3139:
---
[~rkanter] Can you please take a look?
> Oozie validates workflow incorrectly
>
>
> Key: OOZIE-3139
> URL: https://issues.apache.org/jira/browse/OOZIE-3139
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Attachments: workflow.pdf
>
>
> We found that after OOZIE-1978 the attached workflow becomes invalid. Aa we
> found out the following line brings the problem:
> https://github.com/apache/oozie/commit/8e9b9042b3270dc5ff975c44a5c977fcc41250e4#diff-a38ae78886b7c3afb711d32cfb4dbbbdR267
> The following workflow is problematic:
> {noformat}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> Shell action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-3139) Oozie validates workflow incorrectly
[
https://issues.apache.org/jira/browse/OOZIE-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Bodo updated OOZIE-3139:
--
Description:
We found that after OOZIE-1978 the attached workflow becomes invalid. Aa we
found out the following line brings the problem:
https://github.com/apache/oozie/commit/8e9b9042b3270dc5ff975c44a5c977fcc41250e4#diff-a38ae78886b7c3afb711d32cfb4dbbbdR267
. The parameter "okPath" changed from okPath to true.
The following workflow is problematic:
{noformat}
${jobTracker}
${nameNode}
${myscript}
${myscriptPath}
${jobTracker}
${nameNode}
${myscript}
${myscriptPath}
${jobTracker}
${nameNode}
${myscript}
${myscriptPath}
${jobTracker}
${nameNode}
${myscript}
${myscriptPath}
${jobTracker}
${nameNode}
${myscript}
${myscriptPath}
Shell action failed, error
message[${wf:errorMessage(wf:lastErrorNode())}]
{noformat}
was:
We found that after OOZIE-1978 the attached workflow becomes invalid. Aa we
found out the following line brings the problem:
https://github.com/apache/oozie/commit/8e9b9042b3270dc5ff975c44a5c977fcc41250e4#diff-a38ae78886b7c3afb711d32cfb4dbbbdR267
The following workflow is problematic:
{noformat}
${jobTracker}
${nameNode}
${myscript}
${myscriptPath}
${jobTracker}
${nameNode}
${myscript}
${myscriptPath}
${jobTracker}
${nameNode}
${myscript}
${myscriptPath}
${jobTracker}
${nameNode}
${myscript}
${myscriptPath}
${jobTracker}
${nameNode}
${myscript}
${myscriptPath}
Shell action failed, error
message[${wf:errorMessage(wf:lastErrorNode())}]
{noformat}
> Oozie validates workflow incorrectly
>
>
> Key: OOZIE-3139
> URL: https://issues.apache.org/jira/browse/OOZIE-3139
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Attachments: workflow.pdf
>
>
> We found that after OOZIE-1978 the attached workflow becomes invalid. Aa we
> found out the following line brings the problem:
> https://github.com/apache/oozie/commit/8e9b9042b3270dc5ff975c44a5c977fcc41250e4#diff-a38ae78886b7c3afb711d32cfb4dbbbdR267
> . The parameter "okPath" changed from okPath to true.
> The following workflow is problematic:
> {noformat}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> Shell action failed, error
> message[${wf:errorM
[jira] [Updated] (OOZIE-3139) Oozie validates workflow incorrectly
[
https://issues.apache.org/jira/browse/OOZIE-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Bodo updated OOZIE-3139:
--
Attachment: OOZIE-3139.001.patch
> Oozie validates workflow incorrectly
>
>
> Key: OOZIE-3139
> URL: https://issues.apache.org/jira/browse/OOZIE-3139
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Attachments: OOZIE-3139.001.patch, workflow.pdf
>
>
> We found that after OOZIE-1978 the attached workflow becomes invalid. Aa we
> found out the following line brings the problem:
> https://github.com/apache/oozie/commit/8e9b9042b3270dc5ff975c44a5c977fcc41250e4#diff-a38ae78886b7c3afb711d32cfb4dbbbdR267
> . The parameter "okPath" changed from okPath to true.
> The following workflow is problematic:
> {noformat}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> Shell action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-3139) Oozie validates workflow incorrectly
[
https://issues.apache.org/jira/browse/OOZIE-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Bodo updated OOZIE-3139:
--
Attachment: exception.log
> Oozie validates workflow incorrectly
>
>
> Key: OOZIE-3139
> URL: https://issues.apache.org/jira/browse/OOZIE-3139
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Attachments: OOZIE-3139.001.patch, exception.log, workflow.pdf
>
>
> We found that after OOZIE-1978 the attached workflow becomes invalid. Aa we
> found out the following line brings the problem:
> https://github.com/apache/oozie/commit/8e9b9042b3270dc5ff975c44a5c977fcc41250e4#diff-a38ae78886b7c3afb711d32cfb4dbbbdR267
> . The parameter "okPath" changed from okPath to true.
> The following workflow is problematic:
> {noformat}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> Shell action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-3139) Oozie validates workflow incorrectly
[
https://issues.apache.org/jira/browse/OOZIE-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Bodo updated OOZIE-3139:
--
Attachment: OOZIE-3139.002.patch
> Oozie validates workflow incorrectly
>
>
> Key: OOZIE-3139
> URL: https://issues.apache.org/jira/browse/OOZIE-3139
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Attachments: OOZIE-3139.001.patch, OOZIE-3139.002.patch,
> exception.log, workflow.pdf
>
>
> We found that after OOZIE-1978 the attached workflow becomes invalid. Aa we
> found out the following line brings the problem:
> https://github.com/apache/oozie/commit/8e9b9042b3270dc5ff975c44a5c977fcc41250e4#diff-a38ae78886b7c3afb711d32cfb4dbbbdR267
> . The parameter "okPath" changed from okPath to true.
> The following workflow is problematic:
> {noformat}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> Shell action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-3139) Oozie validates workflow incorrectly
[
https://issues.apache.org/jira/browse/OOZIE-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16278841#comment-16278841
]
Denes Bodo commented on OOZIE-3139:
---
[~pbacsko] thanks for your quick response. I've also uploaded the patch to
https://reviews.apache.org/r/64343/
Also great thanks for [~csomaati] finding this wrong parametrization.
> Oozie validates workflow incorrectly
>
>
> Key: OOZIE-3139
> URL: https://issues.apache.org/jira/browse/OOZIE-3139
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Attachments: OOZIE-3139.001.patch, OOZIE-3139.002.patch,
> exception.log, workflow.pdf
>
>
> We found that after OOZIE-1978 the attached workflow becomes invalid. Aa we
> found out the following line brings the problem:
> https://github.com/apache/oozie/commit/8e9b9042b3270dc5ff975c44a5c977fcc41250e4#diff-a38ae78886b7c3afb711d32cfb4dbbbdR267
> . The parameter "okPath" changed from okPath to true.
> The following workflow is problematic:
> {noformat}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> Shell action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-3139) Oozie validates workflow incorrectly
[
https://issues.apache.org/jira/browse/OOZIE-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Bodo updated OOZIE-3139:
--
Attachment: OOZIE-3139.003.patch
003.patch is addressing Abhishek's ask.
> Oozie validates workflow incorrectly
>
>
> Key: OOZIE-3139
> URL: https://issues.apache.org/jira/browse/OOZIE-3139
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Attachments: OOZIE-3139.001.patch, OOZIE-3139.002.patch,
> OOZIE-3139.003.patch, exception.log, workflow.pdf
>
>
> We found that after OOZIE-1978 the attached workflow becomes invalid. Aa we
> found out the following line brings the problem:
> https://github.com/apache/oozie/commit/8e9b9042b3270dc5ff975c44a5c977fcc41250e4#diff-a38ae78886b7c3afb711d32cfb4dbbbdR267
> . The parameter "okPath" changed from okPath to true.
> The following workflow is problematic:
> {noformat}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> Shell action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-3139) Oozie validates workflow incorrectly
[
https://issues.apache.org/jira/browse/OOZIE-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Bodo updated OOZIE-3139:
--
Attachment: OOZIE-3139.004.patch
[~pbacsko] thanks, I did not noticed that.
> Oozie validates workflow incorrectly
>
>
> Key: OOZIE-3139
> URL: https://issues.apache.org/jira/browse/OOZIE-3139
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Attachments: OOZIE-3139.001.patch, OOZIE-3139.002.patch,
> OOZIE-3139.003.patch, OOZIE-3139.004.patch, exception.log, workflow.pdf
>
>
> We found that after OOZIE-1978 the attached workflow becomes invalid. Aa we
> found out the following line brings the problem:
> https://github.com/apache/oozie/commit/8e9b9042b3270dc5ff975c44a5c977fcc41250e4#diff-a38ae78886b7c3afb711d32cfb4dbbbdR267
> . The parameter "okPath" changed from okPath to true.
> The following workflow is problematic:
> {noformat}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> Shell action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (OOZIE-3139) Oozie validates workflow incorrectly
[
https://issues.apache.org/jira/browse/OOZIE-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16280116#comment-16280116
]
Denes Bodo edited comment on OOZIE-3139 at 12/6/17 12:35 PM:
-
[~pbacsko] thanks, I did not notice that.
was (Author: dionusos):
[~pbacsko] thanks, I did not noticed that.
> Oozie validates workflow incorrectly
>
>
> Key: OOZIE-3139
> URL: https://issues.apache.org/jira/browse/OOZIE-3139
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Attachments: OOZIE-3139.001.patch, OOZIE-3139.002.patch,
> OOZIE-3139.003.patch, OOZIE-3139.004.patch, exception.log, workflow.pdf
>
>
> We found that after OOZIE-1978 the attached workflow becomes invalid. Aa we
> found out the following line brings the problem:
> https://github.com/apache/oozie/commit/8e9b9042b3270dc5ff975c44a5c977fcc41250e4#diff-a38ae78886b7c3afb711d32cfb4dbbbdR267
> . The parameter "okPath" changed from okPath to true.
> The following workflow is problematic:
> {noformat}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> Shell action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-3139) Oozie validates workflow incorrectly
[
https://issues.apache.org/jira/browse/OOZIE-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Bodo updated OOZIE-3139:
--
Attachment: OOZIE-3139.005.patch
Add missing Apache license headers.
> Oozie validates workflow incorrectly
>
>
> Key: OOZIE-3139
> URL: https://issues.apache.org/jira/browse/OOZIE-3139
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Attachments: OOZIE-3139.001.patch, OOZIE-3139.002.patch,
> OOZIE-3139.003.patch, OOZIE-3139.004.patch, OOZIE-3139.005.patch,
> exception.log, workflow.pdf
>
>
> We found that after OOZIE-1978 the attached workflow becomes invalid. Aa we
> found out the following line brings the problem:
> https://github.com/apache/oozie/commit/8e9b9042b3270dc5ff975c44a5c977fcc41250e4#diff-a38ae78886b7c3afb711d32cfb4dbbbdR267
> . The parameter "okPath" changed from okPath to true.
> The following workflow is problematic:
> {noformat}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> Shell action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-3139) Oozie validates workflow incorrectly
[
https://issues.apache.org/jira/browse/OOZIE-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16283416#comment-16283416
]
Denes Bodo commented on OOZIE-3139:
---
Does anybody have any comments? Is it okay to merge? Change is on review board,
also.
> Oozie validates workflow incorrectly
>
>
> Key: OOZIE-3139
> URL: https://issues.apache.org/jira/browse/OOZIE-3139
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Attachments: OOZIE-3139.001.patch, OOZIE-3139.002.patch,
> OOZIE-3139.003.patch, OOZIE-3139.004.patch, OOZIE-3139.005.patch,
> exception.log, workflow.pdf
>
>
> We found that after OOZIE-1978 the attached workflow becomes invalid. Aa we
> found out the following line brings the problem:
> https://github.com/apache/oozie/commit/8e9b9042b3270dc5ff975c44a5c977fcc41250e4#diff-a38ae78886b7c3afb711d32cfb4dbbbdR267
> . The parameter "okPath" changed from okPath to true.
> The following workflow is problematic:
> {noformat}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> Shell action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-3355) Regex based search option for searching workflows in the WFM-View of Ambari
[
https://issues.apache.org/jira/browse/OOZIE-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16638145#comment-16638145
]
Denes Bodo commented on OOZIE-3355:
---
As I understand Ambari's Workflow Manager uses Oozie REST API as is. I am not
sure if this issue can be solved efficiently without Oozie's support; querying
all the job names from Oozie means no impact on Oozie but generating heavy DB
queries and great volume of data transmission over REST.
However, that would be a great feature in Oozie, too, if it supported some
wildcard-like search, for example regex, or simply use the SQL's "%" and "_"
characters. In the latter case, it would have only impact on Oozie but not on
Ambari.
> Regex based search option for searching workflows in the WFM-View of Ambari
> ---
>
> Key: OOZIE-3355
> URL: https://issues.apache.org/jira/browse/OOZIE-3355
> Project: Oozie
> Issue Type: New Feature
> Components: bundle, coordinator, core, tools, ui, workflow
>Affects Versions: 4.2.0
>Reporter: Krishnadevan Purushothaman
>Priority: Critical
>
> *Challenge faced :*
> _{color:#d04437}In the WFM view of ambari, there is no Filter option
> available to search for the Workflows. In order to search for the desired
> workflow, one has to type the full name of workflow,coordinator,bundles else
> it does not return anything which becomes a time-consuming job.{color}_
> *Feature description:*
> _{color:#14892c}There is a need for Regex based filter option in order to
> search the workflows without the need of entering the complete name of the
> workflow,coordinator,bundles rather by just typing the first three letters of
> the work post which it should populate suggestions based on the first three
> letters of the workflow,coordinator,bundles through which we can refine and
> optimize the searching mechanism.{color}_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3355) Regex based search option for searching workflows in the WFM-View of Ambari
[
https://issues.apache.org/jira/browse/OOZIE-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16639515#comment-16639515
]
Denes Bodo commented on OOZIE-3355:
---
[~andras.piros] I have never seen Workflow Manager code, but with [~matijhs] we
will take a look what can be done.
> Regex based search option for searching workflows in the WFM-View of Ambari
> ---
>
> Key: OOZIE-3355
> URL: https://issues.apache.org/jira/browse/OOZIE-3355
> Project: Oozie
> Issue Type: New Feature
> Components: bundle, coordinator, core, tools, ui, workflow
>Affects Versions: 4.2.0
>Reporter: Krishnadevan Purushothaman
>Priority: Critical
>
> *Challenge faced :*
> _{color:#d04437}In the WFM view of ambari, there is no Filter option
> available to search for the Workflows. In order to search for the desired
> workflow, one has to type the full name of workflow,coordinator,bundles else
> it does not return anything which becomes a time-consuming job.{color}_
> *Feature description:*
> _{color:#14892c}There is a need for Regex based filter option in order to
> search the workflows without the need of entering the complete name of the
> workflow,coordinator,bundles rather by just typing the first three letters of
> the work post which it should populate suggestions based on the first three
> letters of the workflow,coordinator,bundles through which we can refine and
> optimize the searching mechanism.{color}_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3120) maven-assembly-plugin fails when bumped from 2.2.1
[
https://issues.apache.org/jira/browse/OOZIE-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Bodo updated OOZIE-3120:
--
Attachment: OOZIE-3120-002.patch
> maven-assembly-plugin fails when bumped from 2.2.1
> --
>
> Key: OOZIE-3120
> URL: https://issues.apache.org/jira/browse/OOZIE-3120
> Project: Oozie
> Issue Type: Task
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Major
> Attachments: OOZIE-3120-001.patch, OOZIE-3120-002.patch
>
>
> maven-assembly plugin 3.1.0 is available, version 2.2.1 is old, with upgrade
> to even 2.2.2 build fails with
> {noformat}
> [INFO] --- maven-assembly-plugin:3.1.0:single (default-cli) @ oozie-main ---
> [INFO] Reading assembly descriptor: src/main/assemblies/empty.xml
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Oozie Main .. FAILURE [ 0.972
> s]
> [INFO] Apache Oozie Client SKIPPED
> [INFO] Apache Oozie Share Lib Oozie ... SKIPPED
> [INFO] Apache Oozie Share Lib HCatalog SKIPPED
> [INFO] Apache Oozie Share Lib Distcp .. SKIPPED
> [INFO] Apache Oozie Core .. SKIPPED
> [INFO] Apache Oozie Share Lib Streaming ... SKIPPED
> [INFO] Apache Oozie Share Lib Pig . SKIPPED
> [INFO] Apache Oozie Share Lib Hive SKIPPED
> [INFO] Apache Oozie Share Lib Hive 2 .. SKIPPED
> [INFO] Apache Oozie Share Lib Sqoop ... SKIPPED
> [INFO] Apache Oozie Examples .. SKIPPED
> [INFO] Apache Oozie Share Lib Spark ... SKIPPED
> [INFO] Apache Oozie Share Lib . SKIPPED
> [INFO] Apache Oozie Docs .. SKIPPED
> [INFO] Apache Oozie WebApp SKIPPED
> [INFO] Apache Oozie Tools . SKIPPED
> [INFO] Apache Oozie MiniOozie . SKIPPED
> [INFO] Apache Oozie Server SKIPPED
> [INFO] Apache Oozie Distro SKIPPED
> [INFO] Apache Oozie ZooKeeper Security Tests .. SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 1.872 s
> [INFO] Finished at: 2017-11-10T13:38:17-05:00
> [INFO] Final Memory: 31M/437M
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-assembly-plugin:3.1.0:single (default-cli) on
> project oozie-main: No formats specified in the execution parameters or the
> assembly descriptor. -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> ERROR, Oozie distro creation failed
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3194) Oozie should set proper permissions to sharelib after upload
[ https://issues.apache.org/jira/browse/OOZIE-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Bodo updated OOZIE-3194: -- Attachment: OOZIE-3194-v3.patch > Oozie should set proper permissions to sharelib after upload > > > Key: OOZIE-3194 > URL: https://issues.apache.org/jira/browse/OOZIE-3194 > Project: Oozie > Issue Type: Bug > Components: tools >Affects Versions: 4.3.0 >Reporter: Denes Bodo >Assignee: Denes Bodo >Priority: Critical > Fix For: 4.3.1 > > Attachments: OOZIE-3194-v1.patch, OOZIE-3194-v2.patch, > OOZIE-3194-v3.patch > > > When user changes HDFS umask that does not support world readability to newly > created files, upgrading sharelib causes failing Oozie jobs. > I suggest set the required permissions to sharelib after upgrading it. Files > to 544 and directories to 755. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3186) Oozie is unable to use configuration linked using jceks://file/...
[
https://issues.apache.org/jira/browse/OOZIE-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Bodo updated OOZIE-3186:
--
Attachment: OOZIE-3186-003.patch
> Oozie is unable to use configuration linked using jceks://file/...
> --
>
> Key: OOZIE-3186
> URL: https://issues.apache.org/jira/browse/OOZIE-3186
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.0.0b1, 5.0.0, 4.3.1
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Labels: usability
> Fix For: trunk
>
> Attachments: OOZIE-3186-001.patch, OOZIE-3186-002.patch,
> OOZIE-3186-003.patch
>
>
> When Oozie is used with Ambari, the next configuration makes Oozie fail to
> start:
> {noformat}
>
> hadoop.security.credential.provider.path
> jceks://file/.../oozie-site.jceks
>
> {noformat}
> Value should have *localjceks://* instead of *jceks://*. But Ambari does not
> let change this value. I propose change the url when Oozie loads it.
>
> Stacktrace, when the issue occurs:
> {code:java}
> org.apache.oozie.service.ServiceException: E0103: Could not load service
> classes, Could not load password for [oozie.service.JPAService.jdbc.password]
> at org.apache.oozie.service.Services.loadServices(Services.java:309) at
> org.apache.oozie.service.Services.init(Services.java:213) at
> org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:46)
> at
> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4276)
> at org.apache.catalina.core.StandardContext.start(StandardContext.java:4779)
> at
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:803)
> at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:780)
> at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583) at
> org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:944) at
> org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:779) at
> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:505) at
> org.apache.catalina.startup.HostConfig.start(HostConfig.java:1322) at
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:325) at
> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
> at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1069) at
> org.apache.catalina.core.StandardHost.start(StandardHost.java:822) at
> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061) at
> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463) at
> org.apache.catalina.core.StandardService.start(StandardService.java:525) at
> org.apache.catalina.core.StandardServer.start(StandardServer.java:761) at
> org.apache.catalina.startup.Catalina.start(Catalina.java:595) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) at
> org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) Caused by:
> java.lang.IllegalArgumentException: Could not load password for
> [oozie.service.JPAService.jdbc.password] at
> org.apache.oozie.service.ConfigurationService.getPassword(ConfigurationService.java:615)
> at
> org.apache.oozie.service.ConfigurationService.getPassword(ConfigurationService.java:602)
> at org.apache.oozie.service.JPAService.init(JPAService.java:147) at
> org.apache.oozie.service.Services.setServiceInternal(Services.java:386) at
> org.apache.oozie.service.Services.setService(Services.java:372) at
> org.apache.oozie.service.Services.loadServices(Services.java:305) ... 26 more
> Caused by: java.lang.reflect.InvocationTargetException at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.oozie.service.ConfigurationService.getPassword(ConfigurationService.java:608)
> ... 31 more Caused by: java.lang.UnsupportedOperationException: Accessing
> local file system is not allowed at
> org.apache.hadoop.fs.RawLocalFileSystem.initialize(RawLocalFileSystem.java:48)
> at org.apache.hadoop.fs.LocalFileSystem.initialize(LocalFileSystem.java:47)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2795) at
> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java
[jira] [Commented] (OOZIE-3218) Oozie Sqoop action with command splits the select clause into multiple parts due to delimiter being space
[
https://issues.apache.org/jira/browse/OOZIE-3218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16734035#comment-16734035
]
Denes Bodo commented on OOZIE-3218:
---
I prefer to handle the quotes. In my opinion using quotes in linux environment
is a common when I need to write whitespaces. Also the implementation could be
more maintainable due to we do not have to watch newly created options/token
borders.
> Oozie Sqoop action with command splits the select clause into multiple parts
> due to delimiter being space
> -
>
> Key: OOZIE-3218
> URL: https://issues.apache.org/jira/browse/OOZIE-3218
> Project: Oozie
> Issue Type: Bug
> Components: action, workflow
>Affects Versions: 3.3.2, 4.1.0, 4.2.0, 4.3.0, 5.0.0
> Environment: Hortonworks Hadoop HDP-2.6.4.x release
> oozie admin -version: Oozie server build version: 4.2.0.2.6.4.0-91
>Reporter: Mahesh Balakrishnan
>Assignee: Mahesh Balakrishnan
>Priority: Major
> Attachments: OOZIE-3218-2.patch, OOZIE-3218-3.patch, OOZIE-3218.patch
>
>
> When running a Oozie Sqoop action which has command with {{--query}} in place
> the query is split into multiple parts causing {{"Unrecognized argument:"}}
> and in-turn fails.
> {code:xml}
>
> ${resourceManager}
> ${nameNode}
> import --verbose --connect jdbc:mysql://test.openstacklocal/db
> --query select * from abc where $CONDITIONS --username test --password test
> --driver com.mysql.jdbc.Driver -m 1
>
>
> {code}
>
> Oozie Launcher logs:
> {noformat}
> Sqoop command arguments :
> import
> --verbose
> --connect
> jdbc:mysql://test.openstacklocal/db
> --query
> "select
> *
> from
> abc
> where
> $CONDITIONS"
> --username
> hive
> --password
>
> --driver
> com.mysql.jdbc.Driver
> -m
> 1
> Fetching child yarn jobs
> tag id : oozie-a1bbe03a0983b9e822d12ae7bb269ee3
> 2791 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to
> ResourceManager at hdp263-3.openstacklocal/172.26.105.248:8050
> Child yarn jobs are found -
> =
> >>> Invoking Sqoop command line now >>>
> 3172 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not
> been set in the environment. Cannot check for additional configuration.
> 3172 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not
> been set in the environment. Cannot check for additional configuration.
> 3218 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version:
> 1.4.6.2.6.4.0-91
> 3218 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version:
> 1.4.6.2.6.4.0-91
> 3287 [main] DEBUG org.apache.sqoop.tool.BaseSqoopTool - Enabled debug
> logging.
> 3287 [main] DEBUG org.apache.sqoop.tool.BaseSqoopTool - Enabled debug
> logging.
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Error parsing
> arguments for import:
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Error parsing
> arguments for import:
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: *
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: *
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: from
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: from
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: abc
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: abc
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: where
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: where
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: $CONDITIONS"
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: $CONDITIONS"
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: --username
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: --username
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: abc
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: abc
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: --password
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: --password
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: abc
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: abc
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> ar
[jira] [Commented] (OOZIE-2949) Escape quotes whitespaces in Sqoop field
[
https://issues.apache.org/jira/browse/OOZIE-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16737161#comment-16737161
]
Denes Bodo commented on OOZIE-2949:
---
In case of changing default behaviour: can we just put a switch into
oozie-default.xml where we can declare which mode we want to use? I think
current users will be afraid of an upgrade due to many possible changes in
their workflow. However, new users shall be forced to use the new way. Does it
sound crazy?
> Escape quotes whitespaces in Sqoop field
> --
>
> Key: OOZIE-2949
> URL: https://issues.apache.org/jira/browse/OOZIE-2949
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 4.3.0
>Reporter: Peter Cseh
>Assignee: Andras Salamon
>Priority: Major
> Fix For: 5.2.0
>
> Attachments: OOZIE-2949-0.patch, OOZIE-2949-01.patch,
> OOZIE-2949-02.patch
>
>
> The current behavior of the Sqoop action is:
> {noformat}
> The Sqoop command can be specified either using the command element or
> multiple arg elements.
> When using the command element, Oozie will split the command on every space
> into multiple arguments.
> When using the arg elements, Oozie will pass each argument value as an
> argument to Sqoop.
> {noformat}
> This prevents the user to simply copy-paste the command worked in the shell
> into the workflow.xml.
> We should split the field by taking quotes into account, similar to
> what OOZIE-2391
> did for the Spark action's field.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-2949) Escape quotes whitespaces in Sqoop field
[
https://issues.apache.org/jira/browse/OOZIE-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16737179#comment-16737179
]
Denes Bodo commented on OOZIE-2949:
---
Code review:
* Does ShellSplitter need to be a static inner class? Isn't a normal class in
a separate file means cleaner code?
* Can you please extract characters into ShellSplitter constants which are
used in the state machine?
* In TestSqoopActionExecutor.java:242 you commented out Java code. Was that
intentional and can be deleted?
> Escape quotes whitespaces in Sqoop field
> --
>
> Key: OOZIE-2949
> URL: https://issues.apache.org/jira/browse/OOZIE-2949
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 4.3.0
>Reporter: Peter Cseh
>Assignee: Andras Salamon
>Priority: Major
> Fix For: 5.2.0
>
> Attachments: OOZIE-2949-0.patch, OOZIE-2949-01.patch,
> OOZIE-2949-02.patch
>
>
> The current behavior of the Sqoop action is:
> {noformat}
> The Sqoop command can be specified either using the command element or
> multiple arg elements.
> When using the command element, Oozie will split the command on every space
> into multiple arguments.
> When using the arg elements, Oozie will pass each argument value as an
> argument to Sqoop.
> {noformat}
> This prevents the user to simply copy-paste the command worked in the shell
> into the workflow.xml.
> We should split the field by taking quotes into account, similar to
> what OOZIE-2391
> did for the Spark action's field.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-2689) Spark options --keytab and --principal is not working from Spark Action
[
https://issues.apache.org/jira/browse/OOZIE-2689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16769031#comment-16769031
]
Denes Bodo commented on OOZIE-2689:
---
Is it still a valid ticket? I think it works in 4.3, 4.3.1 and 5.x.
[~andras.piros] [~gezapeti]
> Spark options --keytab and --principal is not working from Spark Action
> ---
>
> Key: OOZIE-2689
> URL: https://issues.apache.org/jira/browse/OOZIE-2689
> Project: Oozie
> Issue Type: Bug
>Reporter: Peter Cseh
>Priority: Major
>
> Spark job running longer than Kerberos ticket lifetime are failing because
> the --principal and --keytab options are not working in Spark Action.
> We're getting messages like
> {quote}
> Delegation Token can be issued only with kerberos or web authentication
> {quote}
> It's possible to work around these issue using the Shell Action and giving
> these options to spark-submit, but it's not a good thing to do.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3326) Sqoop Action should support tez delegation tokens for hive-import
[ https://issues.apache.org/jira/browse/OOZIE-3326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16769103#comment-16769103 ] Denes Bodo commented on OOZIE-3326: --- [~gezapeti] what do you think about this change? In production env the change has been tested. Thanks. > Sqoop Action should support tez delegation tokens for hive-import > - > > Key: OOZIE-3326 > URL: https://issues.apache.org/jira/browse/OOZIE-3326 > Project: Oozie > Issue Type: Bug > Components: action >Reporter: Brian Goerlitz >Assignee: Brian Goerlitz >Priority: Major > Attachments: OZIE-3326_001.patch, OZIE-3326_002.patch > > > SqoopMain needs to support tez delegation tokens for hive-imports. > Implementation is similar to that of HiveMain and Hive2Main. > At present, hive-import will fail to start a tez session in secure > environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3326) Sqoop Action should support tez delegation tokens for hive-import
[ https://issues.apache.org/jira/browse/OOZIE-3326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Bodo updated OOZIE-3326: -- Attachment: OOZIE-3326_003.patch > Sqoop Action should support tez delegation tokens for hive-import > - > > Key: OOZIE-3326 > URL: https://issues.apache.org/jira/browse/OOZIE-3326 > Project: Oozie > Issue Type: Bug > Components: action >Reporter: Brian Goerlitz >Assignee: Brian Goerlitz >Priority: Major > Attachments: OOZIE-3326_003.patch, OZIE-3326_001.patch, > OZIE-3326_002.patch > > > SqoopMain needs to support tez delegation tokens for hive-imports. > Implementation is similar to that of HiveMain and Hive2Main. > At present, hive-import will fail to start a tez session in secure > environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3326) Sqoop Action should support tez delegation tokens for hive-import
[ https://issues.apache.org/jira/browse/OOZIE-3326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Bodo updated OOZIE-3326: -- Attachment: OOZIE-3326_004.patch > Sqoop Action should support tez delegation tokens for hive-import > - > > Key: OOZIE-3326 > URL: https://issues.apache.org/jira/browse/OOZIE-3326 > Project: Oozie > Issue Type: Bug > Components: action >Reporter: Brian Goerlitz >Assignee: Brian Goerlitz >Priority: Major > Attachments: OOZIE-3326_003.patch, OOZIE-3326_004.patch, > OZIE-3326_001.patch, OZIE-3326_002.patch > > > SqoopMain needs to support tez delegation tokens for hive-imports. > Implementation is similar to that of HiveMain and Hive2Main. > At present, hive-import will fail to start a tez session in secure > environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3326) Sqoop Action should support tez delegation tokens for hive-import
[ https://issues.apache.org/jira/browse/OOZIE-3326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Bodo updated OOZIE-3326: -- Attachment: OOZIE-3326_005.patch > Sqoop Action should support tez delegation tokens for hive-import > - > > Key: OOZIE-3326 > URL: https://issues.apache.org/jira/browse/OOZIE-3326 > Project: Oozie > Issue Type: Bug > Components: action >Reporter: Brian Goerlitz >Assignee: Brian Goerlitz >Priority: Major > Attachments: OOZIE-3326_003.patch, OOZIE-3326_004.patch, > OOZIE-3326_005.patch, OZIE-3326_001.patch, OZIE-3326_002.patch > > > SqoopMain needs to support tez delegation tokens for hive-imports. > Implementation is similar to that of HiveMain and Hive2Main. > At present, hive-import will fail to start a tez session in secure > environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3326) Sqoop Action should support tez delegation tokens for hive-import
[ https://issues.apache.org/jira/browse/OOZIE-3326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16770834#comment-16770834 ] Denes Bodo commented on OOZIE-3326: --- I managed to fix the issue found by findbugs and updated the patches. Unfortunately I cannot see any review on review board. Have you pushed the "publish" button? > Sqoop Action should support tez delegation tokens for hive-import > - > > Key: OOZIE-3326 > URL: https://issues.apache.org/jira/browse/OOZIE-3326 > Project: Oozie > Issue Type: Bug > Components: action >Reporter: Brian Goerlitz >Assignee: Brian Goerlitz >Priority: Major > Attachments: OOZIE-3326_003.patch, OOZIE-3326_004.patch, > OOZIE-3326_005.patch, OZIE-3326_001.patch, OZIE-3326_002.patch > > > SqoopMain needs to support tez delegation tokens for hive-imports. > Implementation is similar to that of HiveMain and Hive2Main. > At present, hive-import will fail to start a tez session in secure > environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3326) Sqoop Action should support tez delegation tokens for hive-import
[ https://issues.apache.org/jira/browse/OOZIE-3326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Bodo updated OOZIE-3326: -- Attachment: OOZIE-3326_006.patch > Sqoop Action should support tez delegation tokens for hive-import > - > > Key: OOZIE-3326 > URL: https://issues.apache.org/jira/browse/OOZIE-3326 > Project: Oozie > Issue Type: Bug > Components: action >Reporter: Brian Goerlitz >Assignee: Brian Goerlitz >Priority: Major > Attachments: OOZIE-3326_003.patch, OOZIE-3326_004.patch, > OOZIE-3326_005.patch, OOZIE-3326_006.patch, OZIE-3326_001.patch, > OZIE-3326_002.patch > > > SqoopMain needs to support tez delegation tokens for hive-imports. > Implementation is similar to that of HiveMain and Hive2Main. > At present, hive-import will fail to start a tez session in secure > environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3326) Sqoop Action should support tez delegation tokens for hive-import
[ https://issues.apache.org/jira/browse/OOZIE-3326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Bodo updated OOZIE-3326: -- Attachment: OOZIE-3326_007.patch > Sqoop Action should support tez delegation tokens for hive-import > - > > Key: OOZIE-3326 > URL: https://issues.apache.org/jira/browse/OOZIE-3326 > Project: Oozie > Issue Type: Bug > Components: action >Reporter: Brian Goerlitz >Assignee: Brian Goerlitz >Priority: Major > Attachments: OOZIE-3326_003.patch, OOZIE-3326_004.patch, > OOZIE-3326_005.patch, OOZIE-3326_006.patch, OOZIE-3326_007.patch, > OZIE-3326_001.patch, OZIE-3326_002.patch > > > SqoopMain needs to support tez delegation tokens for hive-imports. > Implementation is similar to that of HiveMain and Hive2Main. > At present, hive-import will fail to start a tez session in secure > environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3326) Sqoop Action should support tez delegation tokens for hive-import
[ https://issues.apache.org/jira/browse/OOZIE-3326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16771626#comment-16771626 ] Denes Bodo commented on OOZIE-3326: --- Thanks [~asalamon74] for your comments and help. > Sqoop Action should support tez delegation tokens for hive-import > - > > Key: OOZIE-3326 > URL: https://issues.apache.org/jira/browse/OOZIE-3326 > Project: Oozie > Issue Type: Bug > Components: action >Reporter: Brian Goerlitz >Assignee: Brian Goerlitz >Priority: Major > Attachments: OOZIE-3326_003.patch, OOZIE-3326_004.patch, > OOZIE-3326_005.patch, OOZIE-3326_006.patch, OOZIE-3326_007.patch, > OZIE-3326_001.patch, OZIE-3326_002.patch > > > SqoopMain needs to support tez delegation tokens for hive-imports. > Implementation is similar to that of HiveMain and Hive2Main. > At present, hive-import will fail to start a tez session in secure > environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (OOZIE-3440) Oozie Spark action replaces local path symlink # to %23
Denes Bodo created OOZIE-3440:
-
Summary: Oozie Spark action replaces local path symlink # to %23
Key: OOZIE-3440
URL: https://issues.apache.org/jira/browse/OOZIE-3440
Project: Oozie
Issue Type: Bug
Components: action
Affects Versions: 5.1.0, 4.3.1
Reporter: Denes Bodo
Assignee: Denes Bodo
When we provide in a hive action:
* --files /etc/spark2/conf/hive-site.xml#hive-site.xml or
* /etc/spark2/conf/hive-site.xml#hive-site.xml or
* /etc/spark2/conf/hive-site.xml#hive-site.xml
we get the following error:
{code}
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain],
main() threw exception, File
file:/etc/spark2/conf/hive-site.xml%23hive-site.xml does not exist
{code}
The culprit seems to be
https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java#L480L489
.
Please help me confirm if this is a bug or not. Meanwhile I am creating a
fix/workaround to this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3440) Oozie Spark action replaces path symlink # to %23
[
https://issues.apache.org/jira/browse/OOZIE-3440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16774908#comment-16774908
]
Denes Bodo commented on OOZIE-3440:
---
Hallo [~nobigo],
thanks for your answer. We tried hdfs://, I've just extended the description.
This fails too.
> Oozie Spark action replaces path symlink # to %23
> -
>
> Key: OOZIE-3440
> URL: https://issues.apache.org/jira/browse/OOZIE-3440
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: 4.3.1, 5.1.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Major
>
> When we provide in a hive action:
> * --files /etc/spark2/conf/hive-site.xml#hive-site.xml or
> * /etc/spark2/conf/hive-site.xml#hive-site.xml or
> * hdfs:///tmp/spark2/conf/hive-site.xml#hive-site.xml or
> * /etc/spark2/conf/hive-site.xml#hive-site.xml
> we get the following error:
> {code}
> Failing Oozie Launcher, Main class
> [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, File
> file:/etc/spark2/conf/hive-site.xml%23hive-site.xml does not exist
> {code}
> The culprit seems to be
> https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java#L480L489
> .
> Please help me confirm if this is a bug or not. Meanwhile I am creating a
> fix/workaround to this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3440) Oozie Spark action replaces path symlink # to %23
[
https://issues.apache.org/jira/browse/OOZIE-3440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Bodo updated OOZIE-3440:
--
Description:
When we provide in a hive action:
* --files /etc/spark2/conf/hive-site.xml#hive-site.xml or
* /etc/spark2/conf/hive-site.xml#hive-site.xml or
* hdfs:///tmp/spark2/conf/hive-site.xml#hive-site.xml or
* /etc/spark2/conf/hive-site.xml#hive-site.xml
we get the following error:
{code}
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain],
main() threw exception, File
file:/etc/spark2/conf/hive-site.xml%23hive-site.xml does not exist
{code}
The culprit seems to be
https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java#L480L489
.
Please help me confirm if this is a bug or not. Meanwhile I am creating a
fix/workaround to this.
was:
When we provide in a hive action:
* --files /etc/spark2/conf/hive-site.xml#hive-site.xml or
* /etc/spark2/conf/hive-site.xml#hive-site.xml or
* /etc/spark2/conf/hive-site.xml#hive-site.xml
we get the following error:
{code}
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain],
main() threw exception, File
file:/etc/spark2/conf/hive-site.xml%23hive-site.xml does not exist
{code}
The culprit seems to be
https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java#L480L489
.
Please help me confirm if this is a bug or not. Meanwhile I am creating a
fix/workaround to this.
Summary: Oozie Spark action replaces path symlink # to %23 (was: Oozie
Spark action replaces local path symlink # to %23)
> Oozie Spark action replaces path symlink # to %23
> -
>
> Key: OOZIE-3440
> URL: https://issues.apache.org/jira/browse/OOZIE-3440
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: 4.3.1, 5.1.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Major
>
> When we provide in a hive action:
> * --files /etc/spark2/conf/hive-site.xml#hive-site.xml or
> * /etc/spark2/conf/hive-site.xml#hive-site.xml or
> * hdfs:///tmp/spark2/conf/hive-site.xml#hive-site.xml or
> * /etc/spark2/conf/hive-site.xml#hive-site.xml
> we get the following error:
> {code}
> Failing Oozie Launcher, Main class
> [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, File
> file:/etc/spark2/conf/hive-site.xml%23hive-site.xml does not exist
> {code}
> The culprit seems to be
> https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java#L480L489
> .
> Please help me confirm if this is a bug or not. Meanwhile I am creating a
> fix/workaround to this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3440) Oozie Spark action replaces path symlink # to %23
[
https://issues.apache.org/jira/browse/OOZIE-3440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Bodo updated OOZIE-3440:
--
Attachment: OOZIE-3440-001.patch
> Oozie Spark action replaces path symlink # to %23
> -
>
> Key: OOZIE-3440
> URL: https://issues.apache.org/jira/browse/OOZIE-3440
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: 4.3.1, 5.1.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Major
> Attachments: OOZIE-3440-001.patch
>
>
> When we provide in a hive action:
> * --files /etc/spark2/conf/hive-site.xml#hive-site.xml or
> * /etc/spark2/conf/hive-site.xml#hive-site.xml or
> * hdfs:///tmp/spark2/conf/hive-site.xml#hive-site.xml or
> * /etc/spark2/conf/hive-site.xml#hive-site.xml
> we get the following error:
> {code}
> Failing Oozie Launcher, Main class
> [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, File
> file:/etc/spark2/conf/hive-site.xml%23hive-site.xml does not exist
> {code}
> The culprit seems to be
> https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java#L480L489
> .
> Please help me confirm if this is a bug or not. Meanwhile I am creating a
> fix/workaround to this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3440) Oozie Spark action replaces path symlink # to %23
[
https://issues.apache.org/jira/browse/OOZIE-3440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16775146#comment-16775146
]
Denes Bodo commented on OOZIE-3440:
---
[~nobigo] have you used spark 1 or spark 2? Can you please share the Hadoop
version too? Thanks
> Oozie Spark action replaces path symlink # to %23
> -
>
> Key: OOZIE-3440
> URL: https://issues.apache.org/jira/browse/OOZIE-3440
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: 4.3.1, 5.1.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Major
> Attachments: OOZIE-3440-001.patch
>
>
> When we provide in a hive action:
> * --files /etc/spark2/conf/hive-site.xml#hive-site.xml or
> * /etc/spark2/conf/hive-site.xml#hive-site.xml or
> * hdfs:///tmp/spark2/conf/hive-site.xml#hive-site.xml or
> * /etc/spark2/conf/hive-site.xml#hive-site.xml
> we get the following error:
> {code}
> Failing Oozie Launcher, Main class
> [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, File
> file:/etc/spark2/conf/hive-site.xml%23hive-site.xml does not exist
> {code}
> The culprit seems to be
> https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java#L480L489
> .
> Please help me confirm if this is a bug or not. Meanwhile I am creating a
> fix/workaround to this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3440) Oozie Spark action replaces path symlink # to %23
[
https://issues.apache.org/jira/browse/OOZIE-3440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Bodo updated OOZIE-3440:
--
Environment:
Hadoop 3
Spark 2 - 2.3.0
> Oozie Spark action replaces path symlink # to %23
> -
>
> Key: OOZIE-3440
> URL: https://issues.apache.org/jira/browse/OOZIE-3440
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: 4.3.1, 5.1.0
> Environment: Hadoop 3
> Spark 2 - 2.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Major
> Attachments: OOZIE-3440-001.patch
>
>
> When we provide in a hive action:
> * --files /etc/spark2/conf/hive-site.xml#hive-site.xml or
> * /etc/spark2/conf/hive-site.xml#hive-site.xml or
> * hdfs:///tmp/spark2/conf/hive-site.xml#hive-site.xml or
> * /etc/spark2/conf/hive-site.xml#hive-site.xml
> we get the following error:
> {code}
> Failing Oozie Launcher, Main class
> [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, File
> file:/etc/spark2/conf/hive-site.xml%23hive-site.xml does not exist
> {code}
> The culprit seems to be
> https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java#L480L489
> .
> Please help me confirm if this is a bug or not. Meanwhile I am creating a
> fix/workaround to this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (OOZIE-3440) Oozie Spark action replaces path symlink # to %23
[
https://issues.apache.org/jira/browse/OOZIE-3440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16775146#comment-16775146
]
Denes Bodo edited comment on OOZIE-3440 at 2/22/19 1:39 PM:
[~nobigo] have you used spark 1 or spark 2? Can you please share the Hadoop
version too? Thanks
cc [~Hari Matta]
was (Author: dionusos):
[~nobigo] have you used spark 1 or spark 2? Can you please share the Hadoop
version too? Thanks
> Oozie Spark action replaces path symlink # to %23
> -
>
> Key: OOZIE-3440
> URL: https://issues.apache.org/jira/browse/OOZIE-3440
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: 4.3.1, 5.1.0
> Environment: Hadoop 3
> Spark 2 - 2.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Major
> Attachments: OOZIE-3440-001.patch
>
>
> When we provide in a hive action:
> * --files /etc/spark2/conf/hive-site.xml#hive-site.xml or
> * /etc/spark2/conf/hive-site.xml#hive-site.xml or
> * hdfs:///tmp/spark2/conf/hive-site.xml#hive-site.xml or
> * /etc/spark2/conf/hive-site.xml#hive-site.xml
> we get the following error:
> {code}
> Failing Oozie Launcher, Main class
> [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, File
> file:/etc/spark2/conf/hive-site.xml%23hive-site.xml does not exist
> {code}
> The culprit seems to be
> https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java#L480L489
> .
> Please help me confirm if this is a bug or not. Meanwhile I am creating a
> fix/workaround to this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3440) Oozie Spark action replaces path symlink # to %23
[
https://issues.apache.org/jira/browse/OOZIE-3440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Bodo updated OOZIE-3440:
--
Environment:
Hadoop 2.7.3
Spark 2 - 2.3.0
was:
Hadoop 3
Spark 2 - 2.3.0
> Oozie Spark action replaces path symlink # to %23
> -
>
> Key: OOZIE-3440
> URL: https://issues.apache.org/jira/browse/OOZIE-3440
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: 4.3.1, 5.1.0
> Environment: Hadoop 2.7.3
> Spark 2 - 2.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Major
> Attachments: OOZIE-3440-001.patch
>
>
> When we provide in a hive action:
> * --files /etc/spark2/conf/hive-site.xml#hive-site.xml or
> * /etc/spark2/conf/hive-site.xml#hive-site.xml or
> * hdfs:///tmp/spark2/conf/hive-site.xml#hive-site.xml or
> * /etc/spark2/conf/hive-site.xml#hive-site.xml
> we get the following error:
> {code}
> Failing Oozie Launcher, Main class
> [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, File
> file:/etc/spark2/conf/hive-site.xml%23hive-site.xml does not exist
> {code}
> The culprit seems to be
> https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java#L480L489
> .
> Please help me confirm if this is a bug or not. Meanwhile I am creating a
> fix/workaround to this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3440) Oozie Spark action replaces path symlink # to %23
[
https://issues.apache.org/jira/browse/OOZIE-3440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16775154#comment-16775154
]
Denes Bodo commented on OOZIE-3440:
---
I've updated the environment info:
HDP 2.6.5, Hadoop 2.7.3 and Spark 2 . 2.3.0 . Oozie parameters the Spark
command's --files option with the "%23" contained string. This can be
reproduces using unit tests.
> Oozie Spark action replaces path symlink # to %23
> -
>
> Key: OOZIE-3440
> URL: https://issues.apache.org/jira/browse/OOZIE-3440
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: 4.3.1, 5.1.0
> Environment: Hadoop 2.7.3
> Spark 2 - 2.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Major
> Attachments: OOZIE-3440-001.patch
>
>
> When we provide in a hive action:
> * --files /etc/spark2/conf/hive-site.xml#hive-site.xml or
> * /etc/spark2/conf/hive-site.xml#hive-site.xml or
> * hdfs:///tmp/spark2/conf/hive-site.xml#hive-site.xml or
> * /etc/spark2/conf/hive-site.xml#hive-site.xml
> we get the following error:
> {code}
> Failing Oozie Launcher, Main class
> [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, File
> file:/etc/spark2/conf/hive-site.xml%23hive-site.xml does not exist
> {code}
> The culprit seems to be
> https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java#L480L489
> .
> Please help me confirm if this is a bug or not. Meanwhile I am creating a
> fix/workaround to this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3440) Oozie Spark action replaces path symlink # to %23
[
https://issues.apache.org/jira/browse/OOZIE-3440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Bodo updated OOZIE-3440:
--
Attachment: OOZIE-3440-002.patch
> Oozie Spark action replaces path symlink # to %23
> -
>
> Key: OOZIE-3440
> URL: https://issues.apache.org/jira/browse/OOZIE-3440
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: 4.3.1, 5.1.0
> Environment: Hadoop 2.7.3
> Spark 2 - 2.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Major
> Attachments: OOZIE-3440-001.patch, OOZIE-3440-002.patch
>
>
> When we provide in a hive action:
> * --files /etc/spark2/conf/hive-site.xml#hive-site.xml or
> * /etc/spark2/conf/hive-site.xml#hive-site.xml or
> * hdfs:///tmp/spark2/conf/hive-site.xml#hive-site.xml or
> * /etc/spark2/conf/hive-site.xml#hive-site.xml
> we get the following error:
> {code}
> Failing Oozie Launcher, Main class
> [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, File
> file:/etc/spark2/conf/hive-site.xml%23hive-site.xml does not exist
> {code}
> The culprit seems to be
> https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java#L480L489
> .
> Please help me confirm if this is a bug or not. Meanwhile I am creating a
> fix/workaround to this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3440) Oozie Spark action replaces path symlink # to %23
[
https://issues.apache.org/jira/browse/OOZIE-3440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777664#comment-16777664
]
Denes Bodo commented on OOZIE-3440:
---
Thanks [~asalamon74] for your comments, and thanks [~nobigo] for the testing.
I've made the requested changes, please see the review board. Thanks.
> Oozie Spark action replaces path symlink # to %23
> -
>
> Key: OOZIE-3440
> URL: https://issues.apache.org/jira/browse/OOZIE-3440
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: 4.3.1, 5.1.0
> Environment: Hadoop 2.7.3
> Spark 2 - 2.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Major
> Attachments: OOZIE-3440-001.patch, OOZIE-3440-002.patch
>
>
> When we provide in a hive action:
> * --files /etc/spark2/conf/hive-site.xml#hive-site.xml or
> * /etc/spark2/conf/hive-site.xml#hive-site.xml or
> * hdfs:///tmp/spark2/conf/hive-site.xml#hive-site.xml or
> * /etc/spark2/conf/hive-site.xml#hive-site.xml
> we get the following error:
> {code}
> Failing Oozie Launcher, Main class
> [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, File
> file:/etc/spark2/conf/hive-site.xml%23hive-site.xml does not exist
> {code}
> The culprit seems to be
> https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java#L480L489
> .
> Please help me confirm if this is a bug or not. Meanwhile I am creating a
> fix/workaround to this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3440) Oozie Spark action replaces path symlink # to %23
[
https://issues.apache.org/jira/browse/OOZIE-3440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Bodo updated OOZIE-3440:
--
Attachment: OOZIE-3440-003.patch
> Oozie Spark action replaces path symlink # to %23
> -
>
> Key: OOZIE-3440
> URL: https://issues.apache.org/jira/browse/OOZIE-3440
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: 4.3.1, 5.1.0
> Environment: Hadoop 2.7.3
> Spark 2 - 2.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Major
> Attachments: OOZIE-3440-001.patch, OOZIE-3440-002.patch,
> OOZIE-3440-003.patch
>
>
> When we provide in a hive action:
> * --files /etc/spark2/conf/hive-site.xml#hive-site.xml or
> * /etc/spark2/conf/hive-site.xml#hive-site.xml or
> * hdfs:///tmp/spark2/conf/hive-site.xml#hive-site.xml or
> * /etc/spark2/conf/hive-site.xml#hive-site.xml
> we get the following error:
> {code}
> Failing Oozie Launcher, Main class
> [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, File
> file:/etc/spark2/conf/hive-site.xml%23hive-site.xml does not exist
> {code}
> The culprit seems to be
> https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java#L480L489
> .
> Please help me confirm if this is a bug or not. Meanwhile I am creating a
> fix/workaround to this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-1702) Reduce warnings thrown while building
[ https://issues.apache.org/jira/browse/OOZIE-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Bodo updated OOZIE-1702: -- Attachment: OOZIE-1702-001.patch > Reduce warnings thrown while building > - > > Key: OOZIE-1702 > URL: https://issues.apache.org/jira/browse/OOZIE-1702 > Project: Oozie > Issue Type: Wish > Components: build >Affects Versions: trunk >Reporter: Mona Chitnis >Assignee: Denes Bodo >Priority: Minor > Labels: documentation, newbie > Attachments: OOZIE-1702-001.patch > > > A lot of warnings are thrown during Oozie compilation, complaining about > javadoc mistakes, missing links etc among probably other severe ones. This > clutters the output. This JIRA is to fix these warnings -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (OOZIE-1702) Reduce warnings thrown while building
[ https://issues.apache.org/jira/browse/OOZIE-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Bodo reassigned OOZIE-1702: - Assignee: Denes Bodo > Reduce warnings thrown while building > - > > Key: OOZIE-1702 > URL: https://issues.apache.org/jira/browse/OOZIE-1702 > Project: Oozie > Issue Type: Wish > Components: build >Affects Versions: trunk >Reporter: Mona Chitnis >Assignee: Denes Bodo >Priority: Minor > Labels: documentation, newbie > > A lot of warnings are thrown during Oozie compilation, complaining about > javadoc mistakes, missing links etc among probably other severe ones. This > clutters the output. This JIRA is to fix these warnings -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-1702) [build] Fix Javadoc warnings
[ https://issues.apache.org/jira/browse/OOZIE-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16780646#comment-16780646 ] Denes Bodo commented on OOZIE-1702: --- Purpose of my patch is to eliminate the 100+ javadoc warnings. The interesting thing that I managed to eliminate them by checking *mvn javadoc:javadoc* and all gone. I'll check it later. > [build] Fix Javadoc warnings > > > Key: OOZIE-1702 > URL: https://issues.apache.org/jira/browse/OOZIE-1702 > Project: Oozie > Issue Type: Wish > Components: build >Affects Versions: trunk >Reporter: Mona Chitnis >Assignee: Denes Bodo >Priority: Minor > Labels: documentation, newbie > Attachments: OOZIE-1702-001.patch > > > A lot of warnings are thrown during Oozie compilation, complaining about > javadoc mistakes, missing links etc among probably other severe ones. This > clutters the output. This JIRA is to fix these warnings -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-1702) [build] Fix Javadoc warnings
[ https://issues.apache.org/jira/browse/OOZIE-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Bodo updated OOZIE-1702: -- Attachment: OOZIE-1702-002.patch > [build] Fix Javadoc warnings > > > Key: OOZIE-1702 > URL: https://issues.apache.org/jira/browse/OOZIE-1702 > Project: Oozie > Issue Type: Wish > Components: build >Affects Versions: trunk >Reporter: Mona Chitnis >Assignee: Denes Bodo >Priority: Minor > Labels: documentation, newbie > Attachments: OOZIE-1702-001.patch, OOZIE-1702-002.patch > > > A lot of warnings are thrown during Oozie compilation, complaining about > javadoc mistakes, missing links etc among probably other severe ones. This > clutters the output. This JIRA is to fix these warnings -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-1974) SSH Action doesn't handle compound commands eg: cmd1 && cmd2 and stuck in [PREP] stage
[
https://issues.apache.org/jira/browse/OOZIE-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16787848#comment-16787848
]
Denes Bodo commented on OOZIE-1974:
---
[~michalisk] Do you think OOZIE-2126 can be related to your issue? Thanks
> SSH Action doesn't handle compound commands eg: cmd1 && cmd2 and stuck in
> [PREP] stage
> --
>
> Key: OOZIE-1974
> URL: https://issues.apache.org/jira/browse/OOZIE-1974
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: trunk
>Reporter: Michalis Kongtongk
>Priority: Major
>
> example WF that will fail:
> {code}
>
>
>
>
> oozie-u...@somedomain.com
> kinit
> oozie-u...@somedomain.com
> -k
> -t
> /home/oozie-user/oozie.keytab
> &&
> hdfs
> dfs
> -put
> /tmp/random-file.txt
> /tmp/random-file.txt
>
>
>
>
>
> Action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> {code}
> Workaround is to execute the compound command in subshell eg: $(cmd1 && cmd2)
> {code}
>
>
>
>
> oozie-u...@somedomain.com
> $(kinit
> oozie-u...@somedomain.com
> -k
> -t
> /home/oozie-user/oozie.keytab
> &&
> hdfs
> dfs
> -put
> /tmp/random-file.txt
> /tmp/random-file.txt
> )
>
>
>
>
>
> Action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> {code}
> Stack trace "org.apache.oozie.command.CommandException: E0800: Action it is
> not running its in [PREP] state,"
> {code}
> 2014-08-05 23:29:49,721 INFO org.apache.oozie.action.ssh.SshActionExecutor:
> SERVER[192-168-88-213.lunix.lan] USER[mko] GROUP[-] TOKEN[] APP[Ssh-copy]
> JOB[008-140805224842389-oozie-oozi-W]
> ACTION[008-140805224842389-oozie-oozi-W@Ssh] start() begins
> 2014-08-05 23:29:49,723 INFO org.apache.oozie.action.ssh.SshActionExecutor:
> SERVER[192-168-88-213.lunix.lan] USER[mko] GROUP[-] TOKEN[] APP[Ssh-copy]
> JOB[008-140805224842389-oozie-oozi-W]
> ACTION[008-140805224842389-oozie-oozi-W@Ssh] Attempting to copy ssh base
> scripts to remote host [m...@192-168-88-213.lunix.lan]
> 2014-08-05 23:29:52,691 INFO org.apache.oozie.servlet.CallbackServlet:
> SERVER[192-168-88-213.lunix.lan] USER[-] GROUP[-] TOKEN[-] APP[-]
> JOB[008-140805224842389-oozie-oozi-W]
> ACTION[008-140805224842389-oozie-oozi-W@Ssh] callback for action
> [008-140805224842389-oozie-oozi-W@Ssh]
> 2014-08-05 23:29:52,714 ERROR
> org.apache.oozie.command.wf.CompletedActionXCommand:
> SERVER[192-168-88-213.lunix.lan] USER[-] GROUP[-] TOKEN[] APP[-]
> JOB[008-140805224842389-oozie-oozi-W]
> ACTION[008-140805224842389-oozie-oozi-W@Ssh] XException,
> org.apache.oozie.command.CommandException: E0800: Action it is not running
> its in [PREP] state, action [008-140805224842389-oozie-oozi-W@Ssh]
> at
> org.apache.oozie.command.wf.CompletedActionXCommand.eagerVerifyPrecondition(CompletedActionXCommand.java:77)
>
> at org.apache.oozie.command.XCommand.call(XCommand.java:251)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:662)
> 2014-08-05 23:29:52,714 WARN
> org.apache.oozie.service.CallableQueueService$CallableWrapper:
> SERVER[192-168-88-213.lunix.lan] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-]
> ACTION[-] exception callable [callback], E0800: Action it is not running its
> in [PREP] state, action [008-140805224842389-oozie-oozi-W@Ssh]
> org.apache.oozie.command.CommandException: E0800: Action it is not running
> its in [PREP] state, action [008-140805224842389-oozie-oozi-W@Ssh]
> at
> org.apache.oozie.command.wf.CompletedActionXCommand.eagerVerifyPrecondition(CompletedActionXCommand.java:77)
>
> at org.apache.oozie.command.XCommand.call(XCommand.java:251)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:662)
> 2014-08-05 23:29:57,262 INFO org.apache.oozie.action.ssh.SshActionExecutor:
> SERVER[192-168-88-213.lunix.lan] USER[mko] GROUP[-] TOKEN[] APP[Ssh-copy]
> JOB[008-140805224842389-oozie-oozi-W]
> ACTION[008-140805224842389-oozie-oozi-W@Ssh] start() ends
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-148) GH-126: PurgeCommand should purge the workflow jobs w/o end_time
[ https://issues.apache.org/jira/browse/OOZIE-148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16797895#comment-16797895 ] Denes Bodo commented on OOZIE-148: -- I think this can be closed. See OOZIE-1401. cc [~BoglarkaEgyed] [~asalamon74] > GH-126: PurgeCommand should purge the workflow jobs w/o end_time > > > Key: OOZIE-148 > URL: https://issues.apache.org/jira/browse/OOZIE-148 > Project: Oozie > Issue Type: Bug >Reporter: Hadoop QA >Priority: Major > > Currently, PurgeCommand is not working with those workflow jobs with > end_time=null. This command needs to take care of those jobs as well. It > could be done by checking created_time if end_time is not available. > The current query: > select w from WorkflowJobBean w where w.endTimestamp < :endTime -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (OOZIE-148) GH-126: PurgeCommand should purge the workflow jobs w/o end_time
[ https://issues.apache.org/jira/browse/OOZIE-148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16797895#comment-16797895 ] Denes Bodo edited comment on OOZIE-148 at 3/21/19 7:51 AM: --- I think this can be closed. See OOZIE-1401. cc [~kmarton] [~asalamon74] was (Author: dionusos): I think this can be closed. See OOZIE-1401. cc [~BoglarkaEgyed] [~asalamon74] > GH-126: PurgeCommand should purge the workflow jobs w/o end_time > > > Key: OOZIE-148 > URL: https://issues.apache.org/jira/browse/OOZIE-148 > Project: Oozie > Issue Type: Bug >Reporter: Hadoop QA >Priority: Major > > Currently, PurgeCommand is not working with those workflow jobs with > end_time=null. This command needs to take care of those jobs as well. It > could be done by checking created_time if end_time is not available. > The current query: > select w from WorkflowJobBean w where w.endTimestamp < :endTime -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (OOZIE-3459) Oozie cannot be built using Java 11
Denes Bodo created OOZIE-3459:
-
Summary: Oozie cannot be built using Java 11
Key: OOZIE-3459
URL: https://issues.apache.org/jira/browse/OOZIE-3459
Project: Oozie
Issue Type: Bug
Components: core, fluent-job
Affects Versions: 5.1.0
Reporter: Denes Bodo
Using OpenJDK 11 I am not able to build Oozie using {{mvn clean install}}.
I found two issues:
* Fluent job API build fails due to Jaxb2 maven plugin.
* No {{com.sun.tools.}} package is available so *TestMetricsInstrumentation*
will not work.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3459) Oozie cannot be built using Java 11
[
https://issues.apache.org/jira/browse/OOZIE-3459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Bodo updated OOZIE-3459:
--
Description:
Using OpenJDK 11 I am not able to build Oozie using {{mvn clean install}}.
I found two issues:
* Fluent job API build fails due to Jaxb2 maven plugin.
* No {{com.sun.tools.}} package is available so *TestMetricsInstrumentation*
will not work.
* Maven surefire plugin has to be updated. It works with 3.0.0-M3
was:
Using OpenJDK 11 I am not able to build Oozie using {{mvn clean install}}.
I found two issues:
* Fluent job API build fails due to Jaxb2 maven plugin.
* No {{com.sun.tools.}} package is available so *TestMetricsInstrumentation*
will not work.
> Oozie cannot be built using Java 11
> ---
>
> Key: OOZIE-3459
> URL: https://issues.apache.org/jira/browse/OOZIE-3459
> Project: Oozie
> Issue Type: Bug
> Components: core, fluent-job
>Affects Versions: 5.1.0
>Reporter: Denes Bodo
>Priority: Major
>
> Using OpenJDK 11 I am not able to build Oozie using {{mvn clean install}}.
> I found two issues:
> * Fluent job API build fails due to Jaxb2 maven plugin.
> * No {{com.sun.tools.}} package is available so *TestMetricsInstrumentation*
> will not work.
> * Maven surefire plugin has to be updated. It works with 3.0.0-M3
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3459) Oozie cannot be built using Java 11
[
https://issues.apache.org/jira/browse/OOZIE-3459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16806797#comment-16806797
]
Denes Bodo commented on OOZIE-3459:
---
[~nobigo] I think JDK11 is good with OpenJPA, according to Unit Tests. However,
I tried Oozie 4.3.1 with setting JAVA_HOME to 11 I got the following:
{noformat}
Error: Could not connect to the database:
org.apache.oozie.service.ServiceException: E0100: Could not initialize service
[org.apache.oozie.service.HadoopAccessorService], failure to login: for
principal: oozie/ctr-e139-1542663976389-92586-01-02.hwx.s...@example.com
from keytab /etc/security/keytabs/oozie.service.keytab
javax.security.auth.login.LoginException: Message stream modified (41)
Stack trace for the error was (for debug purposes):
--
java.lang.Exception: Could not connect to the database:
org.apache.oozie.service.ServiceException: E0100: Could not initialize service
[org.apache.oozie.service.HadoopAccessorService], failure to login: for
principal: oozie/ctr-e139-1542663976389-92586-01-02.hwx.s...@example.com
from keytab /etc/security/keytabs/oozie.service.keytab
javax.security.auth.login.LoginException: Message stream modified (41)
at
org.apache.oozie.tools.OozieDBCLI.validateConnection(OozieDBCLI.java:968)
at org.apache.oozie.tools.OozieDBCLI.createDB(OozieDBCLI.java:193)
at org.apache.oozie.tools.OozieDBCLI.run(OozieDBCLI.java:131)
at org.apache.oozie.tools.OozieDBCLI.main(OozieDBCLI.java:79)
Caused by: org.apache.oozie.service.ServiceException: E0100: Could not
initialize service [org.apache.oozie.service.HadoopAccessorService], failure to
login: for principal:
oozie/ctr-e139-1542663976389-92586-01-02.hwx.s...@example.com from keytab
/etc/security/keytabs/oozie.service.keytab
javax.security.auth.login.LoginException: Message stream modified (41)
at
org.apache.oozie.service.HadoopAccessorService.kerberosInit(HadoopAccessorService.java:244)
at
org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:143)
at
org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:114)
at
org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
at org.apache.oozie.service.Services.setService(Services.java:372)
at org.apache.oozie.service.Services.loadServices(Services.java:305)
at org.apache.oozie.service.Services.init(Services.java:213)
at org.apache.oozie.tools.OozieDBCLI.getJdbcConf(OozieDBCLI.java:180)
at
org.apache.oozie.tools.OozieDBCLI.createConnection(OozieDBCLI.java:956)
at
org.apache.oozie.tools.OozieDBCLI.validateConnection(OozieDBCLI.java:964)
... 3 more
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login:
for principal:
oozie/ctr-e139-1542663976389-92586-01-02.hwx.s...@example.com from keytab
/etc/security/keytabs/oozie.service.keytab
javax.security.auth.login.LoginException: Message stream modified (41)
at
org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1847)
at
org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1215)
at
org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1008)
at
org.apache.oozie.service.HadoopAccessorService.kerberosInit(HadoopAccessorService.java:236)
... 12 more
Caused by: javax.security.auth.login.LoginException: Message stream modified
(41)
at
jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:781)
at
jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:592)
at
java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:726)
at
java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:665)
at
java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:663)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at
java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:663)
at
java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:574)
at
org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1926)
at
org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1837)
... 15 more
Caused by: KrbException: Message stream modified (41)
at
java.security.jgss/sun.security.krb5.KrbKdcRep.check(KrbKdcRep.java:83)
at
java.security.jgss/sun.security.krb5.KrbAsRep.decrypt(KrbAsRep.java:158)
at
java.security.jgss/sun.security.krb5.KrbAsRep.decryptUsi
[jira] [Assigned] (OOZIE-3204) Oozie cannot run HBase code in Java action
[
https://issues.apache.org/jira/browse/OOZIE-3204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Bodo reassigned OOZIE-3204:
-
Assignee: Denes Bodo
> Oozie cannot run HBase code in Java action
> ---
>
> Key: OOZIE-3204
> URL: https://issues.apache.org/jira/browse/OOZIE-3204
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.1
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
>
> After having custom raw file system implementation, HBase (java) action
> cannot run:
> {noformat}
> 018-03-28 06:55:46,372 WARN HbaseCredentials:523 -
> SERVER[ctr-e138-1518143905142-137559-01-03.hwx.site] USER[hrt_qa]
> GROUP[-] TOKEN[] APP[tpch_query1] JOB[002-180328065157516-oozie-oozi-W]
> ACTION[002-180328065157516-oozie-oozi-W@tpch_query1] Exception in
> receiving hbase credentials
> java.io.IOException: java.lang.reflect.InvocationTargetException
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:68)
> at
> org.apache.oozie.action.hadoop.HbaseCredentials$1.run(HbaseCredentials.java:86)
> at
> org.apache.oozie.action.hadoop.HbaseCredentials$1.run(HbaseCredentials.java:84)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:313)
> at
> org.apache.oozie.action.hadoop.HbaseCredentials.obtainToken(HbaseCredentials.java:83)
> at
> org.apache.oozie.action.hadoop.HbaseCredentials.addtoJobConf(HbaseCredentials.java:56)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.setCredentialTokens(JavaActionExecutor.java:1338)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1178)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1424)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65)
> at org.apache.oozie.command.XCommand.call(XCommand.java:287)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:331)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:260)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
> ... 24 more
> Caused by: java.lang.ExceptionInInitializerError
> at org.apache.hadoop.hbase.ClusterId.parseFrom(ClusterId.java:64)
> at
> org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:75)
> at
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:105)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:903)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.(ConnectionManager.java:648)
> ... 29 more
> Caused by: java.lang.UnsupportedOperationException: Accessing local file
> system is not allowed
> at
> org.apache.hadoop.fs.RawLocalFileSystem.initialize(RawLocalFileSystem.java:48)
> at
> org.apache.hadoop.fs.LocalFileSystem.initialize(LocalFileSystem.java:47)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem
[jira] [Resolved] (OOZIE-3204) Oozie cannot run HBase code in Java action
[
https://issues.apache.org/jira/browse/OOZIE-3204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Bodo resolved OOZIE-3204.
---
Resolution: Invalid
> Oozie cannot run HBase code in Java action
> ---
>
> Key: OOZIE-3204
> URL: https://issues.apache.org/jira/browse/OOZIE-3204
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.1
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
>
> After having custom raw file system implementation, HBase (java) action
> cannot run:
> {noformat}
> 018-03-28 06:55:46,372 WARN HbaseCredentials:523 -
> SERVER[ctr-e138-1518143905142-137559-01-03.hwx.site] USER[hrt_qa]
> GROUP[-] TOKEN[] APP[tpch_query1] JOB[002-180328065157516-oozie-oozi-W]
> ACTION[002-180328065157516-oozie-oozi-W@tpch_query1] Exception in
> receiving hbase credentials
> java.io.IOException: java.lang.reflect.InvocationTargetException
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:68)
> at
> org.apache.oozie.action.hadoop.HbaseCredentials$1.run(HbaseCredentials.java:86)
> at
> org.apache.oozie.action.hadoop.HbaseCredentials$1.run(HbaseCredentials.java:84)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:313)
> at
> org.apache.oozie.action.hadoop.HbaseCredentials.obtainToken(HbaseCredentials.java:83)
> at
> org.apache.oozie.action.hadoop.HbaseCredentials.addtoJobConf(HbaseCredentials.java:56)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.setCredentialTokens(JavaActionExecutor.java:1338)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1178)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1424)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65)
> at org.apache.oozie.command.XCommand.call(XCommand.java:287)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:331)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:260)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
> ... 24 more
> Caused by: java.lang.ExceptionInInitializerError
> at org.apache.hadoop.hbase.ClusterId.parseFrom(ClusterId.java:64)
> at
> org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:75)
> at
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:105)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:903)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.(ConnectionManager.java:648)
> ... 29 more
> Caused by: java.lang.UnsupportedOperationException: Accessing local file
> system is not allowed
> at
> org.apache.hadoop.fs.RawLocalFileSystem.initialize(RawLocalFileSystem.java:48)
> at
> org.apache.hadoop.fs.LocalFileSystem.initialize(LocalFileSystem.java:47)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:
[jira] [Commented] (OOZIE-3328) Create Hive compatibility action executor to run hive actions using beeline
[ https://issues.apache.org/jira/browse/OOZIE-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16810778#comment-16810778 ] Denes Bodo commented on OOZIE-3328: --- The suggested implementation needs Hive3 as dependency due to the used beeline-site.xml location. I am uploading the diff here and to [RB|https://reviews.apache.org/r/70406/]. > Create Hive compatibility action executor to run hive actions using beeline > --- > > Key: OOZIE-3328 > URL: https://issues.apache.org/jira/browse/OOZIE-3328 > Project: Oozie > Issue Type: Task > Components: action, core >Affects Versions: 5.0.0, 4.3.1 >Reporter: Denes Bodo >Assignee: Denes Bodo >Priority: Major > Labels: features, usability > > If I am correct then Hive will not support HiveCli for long and Oozie may > have to handle this. > A new executor shall be created which can understand the original hive action > format while this executor shall run the action using beeline. > What are your thoughts? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3328) Create Hive compatibility action executor to run hive actions using beeline
[ https://issues.apache.org/jira/browse/OOZIE-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Bodo updated OOZIE-3328: -- Attachment: OOZIE-3328.001.patch > Create Hive compatibility action executor to run hive actions using beeline > --- > > Key: OOZIE-3328 > URL: https://issues.apache.org/jira/browse/OOZIE-3328 > Project: Oozie > Issue Type: Task > Components: action, core >Affects Versions: 5.0.0, 4.3.1 >Reporter: Denes Bodo >Assignee: Denes Bodo >Priority: Major > Labels: features, usability > Attachments: OOZIE-3328.001.patch > > > If I am correct then Hive will not support HiveCli for long and Oozie may > have to handle this. > A new executor shall be created which can understand the original hive action > format while this executor shall run the action using beeline. > What are your thoughts? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (OOZIE-3328) Create Hive compatibility action executor to run hive actions using beeline
[ https://issues.apache.org/jira/browse/OOZIE-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16810778#comment-16810778 ] Denes Bodo edited comment on OOZIE-3328 at 4/5/19 12:55 PM: The suggested implementation needs Hive3 as dependency due to the used beeline-site.xml location. I am uploading the diff here and to [RB|https://reviews.apache.org/r/70406/]. CC [~matijhs] was (Author: dionusos): The suggested implementation needs Hive3 as dependency due to the used beeline-site.xml location. I am uploading the diff here and to [RB|https://reviews.apache.org/r/70406/]. > Create Hive compatibility action executor to run hive actions using beeline > --- > > Key: OOZIE-3328 > URL: https://issues.apache.org/jira/browse/OOZIE-3328 > Project: Oozie > Issue Type: Task > Components: action, core >Affects Versions: 5.0.0, 4.3.1 >Reporter: Denes Bodo >Assignee: Denes Bodo >Priority: Major > Labels: features, usability > Attachments: OOZIE-3328.001.patch > > > If I am correct then Hive will not support HiveCli for long and Oozie may > have to handle this. > A new executor shall be created which can understand the original hive action > format while this executor shall run the action using beeline. > What are your thoughts? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3459) Oozie cannot be built using Java 11
[
https://issues.apache.org/jira/browse/OOZIE-3459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16810787#comment-16810787
]
Denes Bodo commented on OOZIE-3459:
---
I was not able to build Oozie with openJDK11 due to fluent api's jaxb
dependency.
> Oozie cannot be built using Java 11
> ---
>
> Key: OOZIE-3459
> URL: https://issues.apache.org/jira/browse/OOZIE-3459
> Project: Oozie
> Issue Type: Bug
> Components: core, fluent-job
>Affects Versions: 5.1.0
>Reporter: Denes Bodo
>Priority: Major
>
> Using OpenJDK 11 I am not able to build Oozie using {{mvn clean install}}.
> I found two issues:
> * Fluent job API build fails due to Jaxb2 maven plugin.
> * No {{com.sun.tools.}} package is available so *TestMetricsInstrumentation*
> will not work.
> * Maven surefire plugin has to be updated. It works with 3.0.0-M3
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-2231) Upgrade curator to latest version 2.13.0
[ https://issues.apache.org/jira/browse/OOZIE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815420#comment-16815420 ] Denes Bodo commented on OOZIE-2231: --- [~kmarton] Curator 2.x supports Zookeeper 3.4.x. Curator 3.x supports only Zookeeper 3.5.x. It can be risky from Oozie side to replace the supported ZK version but using the same dependency as Hadoop means reliability. As I see in Hadoop and Hive pom files they use Curator 2.12.0 so Ozoie shall use the same in my opinion. I cannot see if any of these use Curator 3. +1 to Curator 2.12.0 or 2.13.0 > Upgrade curator to latest version 2.13.0 > > > Key: OOZIE-2231 > URL: https://issues.apache.org/jira/browse/OOZIE-2231 > Project: Oozie > Issue Type: Bug > Components: HA >Reporter: Purshotam Shah >Assignee: Julia Kinga Marton >Priority: Blocker > Fix For: 5.2.0 > > Attachments: OOZIE-2231-00.patch, OOZIE-2231-01.patch, > OOZIE-2231-02.patch, OOZIE-2231-02.patch, OOZIE-2231-03.patch, > OOZIE-2231-04.patch, OOZIE-2231-05.patch, OOZIE-2231-06.patch > > > It have some fix related to InterProcessReadWriteLock, ChildReaper, > LeaderSelector which we use. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-2972) Server goes inconsistent when prepare war called with secure without SSL
[
https://issues.apache.org/jira/browse/OOZIE-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818771#comment-16818771
]
Denes Bodo commented on OOZIE-2972:
---
[~asalamon74] Am I right that with eliminating Tomcat and using Jetty instead
this situation is no more possible? Thanks
> Server goes inconsistent when prepare war called with secure without SSL
>
>
> Key: OOZIE-2972
> URL: https://issues.apache.org/jira/browse/OOZIE-2972
> Project: Oozie
> Issue Type: Bug
> Components: security
>Affects Versions: 4.3.0
>Reporter: Prabhu Joseph
>Priority: Major
>
> When prepare-war with secure is called by some user by mistake on a Oozie
> Server which is not configured with SSL causes inconsistent state. Oozie
> Server runs fine but the oozie clients are failed with Authentication failure
> status 302. Checking curl verbose, Oozie Server redirects client to https
> port even though it is not listening. We need to validate the prepare-war
> command when SSL is not configured instead of going to inconsistent state.
> Repro:
> {code}
> Oozie Server without SSL
> /usr/hdp/current/oozie-server/bin/oozie-setup.sh prepare-war -secure
> Start Oozie Server
> curl -ikv -L --negotiate -u:
> http://prabhuzeppelin2.openstacklocal:11000/oozie/v1/admin/status
> * About to connect() to prabhuzeppelin2.openstacklocal port 11000 (#0)
> * Trying 172.26.93.73... connected
> * Connected to prabhuzeppelin2.openstacklocal (172.26.93.73) port 11000 (#0)
> > GET /oozie/v1/admin/status HTTP/1.1
> > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1
> > zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> > Host: prabhuzeppelin2.openstacklocal:11000
> > Accept: */*
> >
> < HTTP/1.1 302 Found
> HTTP/1.1 302 Found
> < Server: Apache-Coyote/1.1
> Server: Apache-Coyote/1.1
> < Pragma: No-cache
> Pragma: No-cache
> < Cache-Control: no-cache
> Cache-Control: no-cache
> < Expires: Thu, 01 Jan 1970 00:00:00 UTC
> Expires: Thu, 01 Jan 1970 00:00:00 UTC
> < Location: https://prabhuzeppelin2.openstacklocal:11443/oozie/v1/admin/status
> Location: https://prabhuzeppelin2.openstacklocal:11443/oozie/v1/admin/status
> < Content-Length: 0
> Content-Length: 0
> < Date: Tue, 27 Jun 2017 11:05:45 GMT
> Date: Tue, 27 Jun 2017 11:05:45 GMT
> <
> * Connection #0 to host prabhuzeppelin2.openstacklocal left intact
> * Issue another request to this URL:
> 'https://prabhuzeppelin2.openstacklocal:11443/oozie/v1/admin/status'
> * About to connect() to prabhuzeppelin2.openstacklocal port 11443 (#1)
> * Trying 172.26.93.73... Connection refused
> * couldn't connect to host
> * Closing connection #1
> curl: (7) couldn't connect to host
> * Closing connection #0
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3459) Oozie cannot be built using Java 11
[
https://issues.apache.org/jira/browse/OOZIE-3459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16834743#comment-16834743
]
Denes Bodo commented on OOZIE-3459:
---
[~kmarton] Thank you for your suggestion and offering. When this ticket was
created I did not foresee such number of issues. It is a good idea to track the
found ones with this ticket as an umbrella.
> Oozie cannot be built using Java 11
> ---
>
> Key: OOZIE-3459
> URL: https://issues.apache.org/jira/browse/OOZIE-3459
> Project: Oozie
> Issue Type: Bug
> Components: core, fluent-job
>Affects Versions: 5.1.0
>Reporter: Denes Bodo
>Priority: Major
>
> Using OpenJDK 11 I am not able to build Oozie using {{mvn clean install}}.
> I found two issues:
> * Fluent job API build fails due to Jaxb2 maven plugin.
> * No {{com.sun.tools.}} package is available so *TestMetricsInstrumentation*
> will not work.
> * Maven surefire plugin has to be updated. It works with 3.0.0-M3
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (OOZIE-3529) Oozie not supported for s3 as filesystem
Denes Bodo created OOZIE-3529:
-
Summary: Oozie not supported for s3 as filesystem
Key: OOZIE-3529
URL: https://issues.apache.org/jira/browse/OOZIE-3529
Project: Oozie
Issue Type: Bug
Components: core
Affects Versions: 5.1.0, 4.3.1
Reporter: Denes Bodo
Attachments: id.pig, job.properties, workflow.xml
Many customer who uses s3 file system as secondary one experiences the
following error when Oozie tries to submit the Yarn application:
{noformat}
2019-04-29 13:02:53,770 WARN ForkedActionStartXCommand:523 -
SERVER[hwnode1.puretec.purestorage.com] USER[hrt_qa] GROUP[-] TOKEN[]
APP[demo-wf] JOB[001-190423141707256-oozie-oozi-W]
ACTION[001-190423141707256-oozie-oozi-W@streaming-node] Error starting
action [streaming-node]. ErrorType [ERROR], ErrorCode
[UnsupportedOperationException], Message [UnsupportedOperationException:
Accessing local file system is not allowed]
org.apache.oozie.action.ActionExecutorException: UnsupportedOperationException:
Accessing local file system is not allowed
at
org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.createLauncherConf(JavaActionExecutor.java:1092)
at
org.apache.oozie.action.hadoop.MapReduceActionExecutor.createLauncherConf(MapReduceActionExecutor.java:309)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1197)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1472)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
at
org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:41)
at
org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:30)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.UnsupportedOperationException: Accessing local file system
is not allowed
at
org.apache.hadoop.fs.RawLocalFileSystem.initialize(RawLocalFileSystem.java:48)
at
org.apache.hadoop.fs.LocalFileSystem.initialize(LocalFileSystem.java:47)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:433)
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:301)
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:378)
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createTmpFileForWrite(LocalDirAllocator.java:461)
at
org.apache.hadoop.fs.LocalDirAllocator.createTmpFileForWrite(LocalDirAllocator.java:200)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.createTmpFileForWrite(S3AFileSystem.java:572)
at
org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory.create(S3ADataBlocks.java:811)
at
org.apache.hadoop.fs.s3a.S3ABlockOutputStream.createBlockIfNeeded(S3ABlockOutputStream.java:190)
at
org.apache.hadoop.fs.s3a.S3ABlockOutputStream.(S3ABlockOutputStream.java:168)
at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:778)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1169)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1038)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1026)
at
org.apache.oozie.action.hadoop.LauncherMapperHelper.setupLauncherInfo(LauncherMapperHelper.java:156)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.createLauncherConf(JavaActionExecutor.java:1033)
... 12 more
{noformat}
Does anybody has any idea how could we modify the RawLocalFileSystem class to
make it a bit less strict?
Thank you for the repro wf to Soumitra Sulav.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (OOZIE-3529) Oozie not supported for s3 as filesystem
[
https://issues.apache.org/jira/browse/OOZIE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891793#comment-16891793
]
Denes Bodo commented on OOZIE-3529:
---
As I experienced mentioning the following two properties in workflow.xml action
configuration, the S3AFileSystem will not use local file system as temp.
{code:xml}
oozie.launcher.fs.s3a.fast.upload.buffer
bytebuffer
oozie.launcher.fs.s3a.impl.disable.cache
true
{code}
Conclusion now:
This can be a workaround if customer wants to have the cve fixed and accept to
modify their workflows.
This cannot be a solution as it requires modification of all the job
configuration.
> Oozie not supported for s3 as filesystem
>
>
> Key: OOZIE-3529
> URL: https://issues.apache.org/jira/browse/OOZIE-3529
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.1, 5.1.0
>Reporter: Denes Bodo
>Priority: Critical
> Labels: S3
> Attachments: id.pig, job.properties, workflow.xml
>
>
> Many customer who uses s3 file system as secondary one experiences the
> following error when Oozie tries to submit the Yarn application:
> {noformat}
> 2019-04-29 13:02:53,770 WARN ForkedActionStartXCommand:523 -
> SERVER[hwnode1.puretec.purestorage.com] USER[hrt_qa] GROUP[-] TOKEN[]
> APP[demo-wf] JOB[001-190423141707256-oozie-oozi-W]
> ACTION[001-190423141707256-oozie-oozi-W@streaming-node] Error starting
> action [streaming-node]. ErrorType [ERROR], ErrorCode
> [UnsupportedOperationException], Message [UnsupportedOperationException:
> Accessing local file system is not allowed]
> org.apache.oozie.action.ActionExecutorException:
> UnsupportedOperationException: Accessing local file system is not allowed
> at
> org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.createLauncherConf(JavaActionExecutor.java:1092)
> at
> org.apache.oozie.action.hadoop.MapReduceActionExecutor.createLauncherConf(MapReduceActionExecutor.java:309)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1197)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1472)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
> at
> org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:41)
> at
> org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:30)
> at org.apache.oozie.command.XCommand.call(XCommand.java:287)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.UnsupportedOperationException: Accessing local file
> system is not allowed
> at
> org.apache.hadoop.fs.RawLocalFileSystem.initialize(RawLocalFileSystem.java:48)
> at
> org.apache.hadoop.fs.LocalFileSystem.initialize(LocalFileSystem.java:47)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
> at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:433)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:301)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:378)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createTmpFileForWrite(LocalDirAllocator.java:461)
> at
> org.apache.hadoop.fs.LocalDirAllocator.createTmpFileForWrite(LocalDirAllocator.java:200)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.createTmpFileForWrite(S3AFileSystem.java:572)
> at
> org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory.create(S3ADataBlocks.java:811)
> at
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream.createBlockIfNeeded(S3ABlockOutputStream.java:190)
> at
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream.
