[jira] [Commented] (OOZIE-1954) Add a way for the MapReduce action to be configured by Java code
[
https://issues.apache.org/jira/browse/OOZIE-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14153721#comment-14153721
]
Mona Chitnis commented on OOZIE-1954:
-
Good work Robert!
> Add a way for the MapReduce action to be configured by Java code
>
>
> Key: OOZIE-1954
> URL: https://issues.apache.org/jira/browse/OOZIE-1954
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: trunk
>
> Attachments: OOZIE-1954.patch, OOZIE-1954.patch, OOZIE-1954.patch
>
>
> With certain other components (e.g. Avro, HFileOutputFormat (HBase), etc), it
> becomes impractical to use the MapReduce action and users must instead use
> the Java action. The problem is that these components require a lot of extra
> configuration that is often hidden from the user in Java code (e.g.
> HFileOutputFormat.configureIncrementalLoad(job, table); which can also
> include decision logic, serialization, and other things that we can't do in
> an XML file directly.
> One way to solve this problem is to allow the user to give the MR action some
> Java code that would do this configuration, similar to how we allow the
> {{}} field to specify an external XML file of configuration
> properties.
> In more detail, we could have an interface; something like this:
> {code}
> public interface OozieActionConfigurator {
> public void updateOozieActionConfiguration(Configuration conf);
> }
> {code}
> that the user can implement, create a jar, and include with their MR action
> (i.e. add a "{{}}" field that let's them specify the class
> name). To protect the Oozie server from running user code (which could do
> anything it wants really), it would have to be run in the Launcher Job. The
> Launcher Job could call this method after it loads the configuration prepared
> by the Oozie server.
> Another thing this will be helpful is with users who use the Java action to
> launch MR jobs and expect a bunch of things to be done for them that are not
> (e.g. delegation token propagation, config loading, returning the hadoop job
> to Oozie, etc). These are all done with the MR action, so the more users we
> can move to the MR action from the Java action, the less they'll run into
> these difficulties.
> Some of this may change slightly as I try to actually implement this (e.g.
> have to handle throwing exceptions etc). And one thing I may do is keep this
> general enough that it should be compatible with all action types in case we
> want to add this to any of them in the future; though for now, the schema
> would only accept it for the MapReduce action.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-1954) Add a way for the MapReduce action to be configured by Java code
[
https://issues.apache.org/jira/browse/OOZIE-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14135397#comment-14135397
]
Hadoop QA commented on OOZIE-1954:
--
Testing JIRA OOZIE-1954
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch does adds/modifies 3 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1530
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:green}*+1 Overall result, good!, no -1s*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/1980/
> Add a way for the MapReduce action to be configured by Java code
>
>
> Key: OOZIE-1954
> URL: https://issues.apache.org/jira/browse/OOZIE-1954
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: OOZIE-1954.patch, OOZIE-1954.patch, OOZIE-1954.patch
>
>
> With certain other components (e.g. Avro, HFileOutputFormat (HBase), etc), it
> becomes impractical to use the MapReduce action and users must instead use
> the Java action. The problem is that these components require a lot of extra
> configuration that is often hidden from the user in Java code (e.g.
> HFileOutputFormat.configureIncrementalLoad(job, table); which can also
> include decision logic, serialization, and other things that we can't do in
> an XML file directly.
> One way to solve this problem is to allow the user to give the MR action some
> Java code that would do this configuration, similar to how we allow the
> {{}} field to specify an external XML file of configuration
> properties.
> In more detail, we could have an interface; something like this:
> {code}
> public interface OozieActionConfigurator {
> public void updateOozieActionConfiguration(Configuration conf);
> }
> {code}
> that the user can implement, create a jar, and include with their MR action
> (i.e. add a "{{}}" field that let's them specify the class
> name). To protect the Oozie server from running user code (which could do
> anything it wants really), it would have to be run in the Launcher Job. The
> Launcher Job could call this method after it loads the configuration prepared
> by the Oozie server.
> Another thing this will be helpful is with users who use the Java action to
> launch MR jobs and expect a bunch of things to be done for them that are not
> (e.g. delegation token propagation, config loading, returning the hadoop job
> to Oozie, etc). These are all done with the MR action, so the more users we
> can move to the MR action from the Java action, the less they'll run into
> these difficulties.
> Some of this may change slightly as I try to actually implement this (e.g.
> have to handle throwing exceptions etc). And one thing I may do is keep this
> general enough that it should be compatible with all action types in case we
> want to add this to any of them in the future; though for now, the schema
> would only accept it for the MapReduce action.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-1954) Add a way for the MapReduce action to be configured by Java code
[
https://issues.apache.org/jira/browse/OOZIE-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14135319#comment-14135319
]
Shwetha G S commented on OOZIE-1954:
Triggered https://builds.apache.org/job/oozie-trunk-precommit-build/1980/
> Add a way for the MapReduce action to be configured by Java code
>
>
> Key: OOZIE-1954
> URL: https://issues.apache.org/jira/browse/OOZIE-1954
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: OOZIE-1954.patch, OOZIE-1954.patch, OOZIE-1954.patch
>
>
> With certain other components (e.g. Avro, HFileOutputFormat (HBase), etc), it
> becomes impractical to use the MapReduce action and users must instead use
> the Java action. The problem is that these components require a lot of extra
> configuration that is often hidden from the user in Java code (e.g.
> HFileOutputFormat.configureIncrementalLoad(job, table); which can also
> include decision logic, serialization, and other things that we can't do in
> an XML file directly.
> One way to solve this problem is to allow the user to give the MR action some
> Java code that would do this configuration, similar to how we allow the
> {{}} field to specify an external XML file of configuration
> properties.
> In more detail, we could have an interface; something like this:
> {code}
> public interface OozieActionConfigurator {
> public void updateOozieActionConfiguration(Configuration conf);
> }
> {code}
> that the user can implement, create a jar, and include with their MR action
> (i.e. add a "{{}}" field that let's them specify the class
> name). To protect the Oozie server from running user code (which could do
> anything it wants really), it would have to be run in the Launcher Job. The
> Launcher Job could call this method after it loads the configuration prepared
> by the Oozie server.
> Another thing this will be helpful is with users who use the Java action to
> launch MR jobs and expect a bunch of things to be done for them that are not
> (e.g. delegation token propagation, config loading, returning the hadoop job
> to Oozie, etc). These are all done with the MR action, so the more users we
> can move to the MR action from the Java action, the less they'll run into
> these difficulties.
> Some of this may change slightly as I try to actually implement this (e.g.
> have to handle throwing exceptions etc). And one thing I may do is keep this
> general enough that it should be compatible with all action types in case we
> want to add this to any of them in the future; though for now, the schema
> would only accept it for the MapReduce action.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-1954) Add a way for the MapReduce action to be configured by Java code
[
https://issues.apache.org/jira/browse/OOZIE-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14135305#comment-14135305
]
Shwetha G S commented on OOZIE-1954:
Yes, +1. Thanks
> Add a way for the MapReduce action to be configured by Java code
>
>
> Key: OOZIE-1954
> URL: https://issues.apache.org/jira/browse/OOZIE-1954
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: OOZIE-1954.patch, OOZIE-1954.patch, OOZIE-1954.patch
>
>
> With certain other components (e.g. Avro, HFileOutputFormat (HBase), etc), it
> becomes impractical to use the MapReduce action and users must instead use
> the Java action. The problem is that these components require a lot of extra
> configuration that is often hidden from the user in Java code (e.g.
> HFileOutputFormat.configureIncrementalLoad(job, table); which can also
> include decision logic, serialization, and other things that we can't do in
> an XML file directly.
> One way to solve this problem is to allow the user to give the MR action some
> Java code that would do this configuration, similar to how we allow the
> {{}} field to specify an external XML file of configuration
> properties.
> In more detail, we could have an interface; something like this:
> {code}
> public interface OozieActionConfigurator {
> public void updateOozieActionConfiguration(Configuration conf);
> }
> {code}
> that the user can implement, create a jar, and include with their MR action
> (i.e. add a "{{}}" field that let's them specify the class
> name). To protect the Oozie server from running user code (which could do
> anything it wants really), it would have to be run in the Launcher Job. The
> Launcher Job could call this method after it loads the configuration prepared
> by the Oozie server.
> Another thing this will be helpful is with users who use the Java action to
> launch MR jobs and expect a bunch of things to be done for them that are not
> (e.g. delegation token propagation, config loading, returning the hadoop job
> to Oozie, etc). These are all done with the MR action, so the more users we
> can move to the MR action from the Java action, the less they'll run into
> these difficulties.
> Some of this may change slightly as I try to actually implement this (e.g.
> have to handle throwing exceptions etc). And one thing I may do is keep this
> general enough that it should be compatible with all action types in case we
> want to add this to any of them in the future; though for now, the schema
> would only accept it for the MapReduce action.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-1954) Add a way for the MapReduce action to be configured by Java code
[
https://issues.apache.org/jira/browse/OOZIE-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14132337#comment-14132337
]
Hadoop QA commented on OOZIE-1954:
--
Testing JIRA OOZIE-1954
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch does adds/modifies 3 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/1975/
> Add a way for the MapReduce action to be configured by Java code
>
>
> Key: OOZIE-1954
> URL: https://issues.apache.org/jira/browse/OOZIE-1954
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: OOZIE-1954.patch, OOZIE-1954.patch
>
>
> With certain other components (e.g. Avro, HFileOutputFormat (HBase), etc), it
> becomes impractical to use the MapReduce action and users must instead use
> the Java action. The problem is that these components require a lot of extra
> configuration that is often hidden from the user in Java code (e.g.
> HFileOutputFormat.configureIncrementalLoad(job, table); which can also
> include decision logic, serialization, and other things that we can't do in
> an XML file directly.
> One way to solve this problem is to allow the user to give the MR action some
> Java code that would do this configuration, similar to how we allow the
> {{}} field to specify an external XML file of configuration
> properties.
> In more detail, we could have an interface; something like this:
> {code}
> public interface OozieActionConfigurator {
> public void updateOozieActionConfiguration(Configuration conf);
> }
> {code}
> that the user can implement, create a jar, and include with their MR action
> (i.e. add a "{{}}" field that let's them specify the class
> name). To protect the Oozie server from running user code (which could do
> anything it wants really), it would have to be run in the Launcher Job. The
> Launcher Job could call this method after it loads the configuration prepared
> by the Oozie server.
> Another thing this will be helpful is with users who use the Java action to
> launch MR jobs and expect a bunch of things to be done for them that are not
> (e.g. delegation token propagation, config loading, returning the hadoop job
> to Oozie, etc). These are all done with the MR action, so the more users we
> can move to the MR action from the Java action, the less they'll run into
> these difficulties.
> Some of this may change slightly as I try to actually implement this (e.g.
> have to handle throwing exceptions etc). And one thing I may do is keep this
> general enough that it should be compatible with all action types in case we
> want to add this to any of them in the future; though for now, the schema
> would only accept it for the MapReduce action.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-1954) Add a way for the MapReduce action to be configured by Java code
[
https://issues.apache.org/jira/browse/OOZIE-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14119259#comment-14119259
]
Hadoop QA commented on OOZIE-1954:
--
Testing JIRA OOZIE-1954
Cleaning local git workspace
{color:red}-1{color} Patch failed to apply to head of branch
> Add a way for the MapReduce action to be configured by Java code
>
>
> Key: OOZIE-1954
> URL: https://issues.apache.org/jira/browse/OOZIE-1954
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: OOZIE-1954.patch
>
>
> With certain other components (e.g. Avro, HFileOutputFormat (HBase), etc), it
> becomes impractical to use the MapReduce action and users must instead use
> the Java action. The problem is that these components require a lot of extra
> configuration that is often hidden from the user in Java code (e.g.
> HFileOutputFormat.configureIncrementalLoad(job, table); which can also
> include decision logic, serialization, and other things that we can't do in
> an XML file directly.
> One way to solve this problem is to allow the user to give the MR action some
> Java code that would do this configuration, similar to how we allow the
> {{}} field to specify an external XML file of configuration
> properties.
> In more detail, we could have an interface; something like this:
> {code}
> public interface OozieActionConfigurator {
> public void updateOozieActionConfiguration(Configuration conf);
> }
> {code}
> that the user can implement, create a jar, and include with their MR action
> (i.e. add a "{{}}" field that let's them specify the class
> name). To protect the Oozie server from running user code (which could do
> anything it wants really), it would have to be run in the Launcher Job. The
> Launcher Job could call this method after it loads the configuration prepared
> by the Oozie server.
> Another thing this will be helpful is with users who use the Java action to
> launch MR jobs and expect a bunch of things to be done for them that are not
> (e.g. delegation token propagation, config loading, returning the hadoop job
> to Oozie, etc). These are all done with the MR action, so the more users we
> can move to the MR action from the Java action, the less they'll run into
> these difficulties.
> Some of this may change slightly as I try to actually implement this (e.g.
> have to handle throwing exceptions etc). And one thing I may do is keep this
> general enough that it should be compatible with all action types in case we
> want to add this to any of them in the future; though for now, the schema
> would only accept it for the MapReduce action.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-1954) Add a way for the MapReduce action to be configured by Java code
[
https://issues.apache.org/jira/browse/OOZIE-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14110095#comment-14110095
]
Hadoop QA commented on OOZIE-1954:
--
Testing JIRA OOZIE-1954
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch does adds/modifies 4 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:red}-1 COMPILE{color}
.{color:red}-1{color} HEAD does not compile
.{color:red}-1{color} patch does not compile
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:red}-1 DISTRO{color}
.{color:red}-1{color} distro tarball fails with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/1708/
> Add a way for the MapReduce action to be configured by Java code
>
>
> Key: OOZIE-1954
> URL: https://issues.apache.org/jira/browse/OOZIE-1954
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: OOZIE-1954.patch
>
>
> With certain other components (e.g. Avro, HFileOutputFormat (HBase), etc), it
> becomes impractical to use the MapReduce action and users must instead use
> the Java action. The problem is that these components require a lot of extra
> configuration that is often hidden from the user in Java code (e.g.
> HFileOutputFormat.configureIncrementalLoad(job, table); which can also
> include decision logic, serialization, and other things that we can't do in
> an XML file directly.
> One way to solve this problem is to allow the user to give the MR action some
> Java code that would do this configuration, similar to how we allow the
> {{}} field to specify an external XML file of configuration
> properties.
> In more detail, we could have an interface; something like this:
> {code}
> public interface OozieActionConfigurator {
> public void updateOozieActionConfiguration(Configuration conf);
> }
> {code}
> that the user can implement, create a jar, and include with their MR action
> (i.e. add a "{{}}" field that let's them specify the class
> name). To protect the Oozie server from running user code (which could do
> anything it wants really), it would have to be run in the Launcher Job. The
> Launcher Job could call this method after it loads the configuration prepared
> by the Oozie server.
> Another thing this will be helpful is with users who use the Java action to
> launch MR jobs and expect a bunch of things to be done for them that are not
> (e.g. delegation token propagation, config loading, returning the hadoop job
> to Oozie, etc). These are all done with the MR action, so the more users we
> can move to the MR action from the Java action, the less they'll run into
> these difficulties.
> Some of this may change slightly as I try to actually implement this (e.g.
> have to handle throwing exceptions etc). And one thing I may do is keep this
> general enough that it should be compatible with all action types in case we
> want to add this to any of them in the future; though for now, the schema
> would only accept it for the MapReduce action.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1954) Add a way for the MapReduce action to be configured by Java code
[
https://issues.apache.org/jira/browse/OOZIE-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14107582#comment-14107582
]
Hadoop QA commented on OOZIE-1954:
--
Testing JIRA OOZIE-1954
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch does adds/modifies 4 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1511
.Tests failed: 1
.Tests errors: 0
.The patch failed the following testcases:
.
testActionKillCommandDate(org.apache.oozie.command.coord.TestCoordActionsKillXCommand)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/1620/
> Add a way for the MapReduce action to be configured by Java code
>
>
> Key: OOZIE-1954
> URL: https://issues.apache.org/jira/browse/OOZIE-1954
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: OOZIE-1954.patch
>
>
> With certain other components (e.g. Avro, HFileOutputFormat (HBase), etc), it
> becomes impractical to use the MapReduce action and users must instead use
> the Java action. The problem is that these components require a lot of extra
> configuration that is often hidden from the user in Java code (e.g.
> HFileOutputFormat.configureIncrementalLoad(job, table); which can also
> include decision logic, serialization, and other things that we can't do in
> an XML file directly.
> One way to solve this problem is to allow the user to give the MR action some
> Java code that would do this configuration, similar to how we allow the
> {{}} field to specify an external XML file of configuration
> properties.
> In more detail, we could have an interface; something like this:
> {code}
> public interface OozieActionConfigurator {
> public void updateOozieActionConfiguration(Configuration conf);
> }
> {code}
> that the user can implement, create a jar, and include with their MR action
> (i.e. add a "{{}}" field that let's them specify the class
> name). To protect the Oozie server from running user code (which could do
> anything it wants really), it would have to be run in the Launcher Job. The
> Launcher Job could call this method after it loads the configuration prepared
> by the Oozie server.
> Another thing this will be helpful is with users who use the Java action to
> launch MR jobs and expect a bunch of things to be done for them that are not
> (e.g. delegation token propagation, config loading, returning the hadoop job
> to Oozie, etc). These are all done with the MR action, so the more users we
> can move to the MR action from the Java action, the less they'll run into
> these difficulties.
> Some of this may change slightly as I try to actually implement this (e.g.
> have to handle throwing exceptions etc). And one thing I may do is keep this
> general enough that it should be compatible with all action types in case we
> want to add this to any of them in the future; though for now, the schema
> would only accept it for the MapReduce action.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1954) Add a way for the MapReduce action to be configured by Java code
[
https://issues.apache.org/jira/browse/OOZIE-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14083639#comment-14083639
]
Hadoop QA commented on OOZIE-1954:
--
Testing JIRA OOZIE-1954
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch does adds/modifies 4 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1508
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:green}*+1 Overall result, good!, no -1s*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/1430/
> Add a way for the MapReduce action to be configured by Java code
>
>
> Key: OOZIE-1954
> URL: https://issues.apache.org/jira/browse/OOZIE-1954
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: OOZIE-1954.patch
>
>
> With certain other components (e.g. Avro, HFileOutputFormat (HBase), etc), it
> becomes impractical to use the MapReduce action and users must instead use
> the Java action. The problem is that these components require a lot of extra
> configuration that is often hidden from the user in Java code (e.g.
> HFileOutputFormat.configureIncrementalLoad(job, table); which can also
> include decision logic, serialization, and other things that we can't do in
> an XML file directly.
> One way to solve this problem is to allow the user to give the MR action some
> Java code that would do this configuration, similar to how we allow the
> {{}} field to specify an external XML file of configuration
> properties.
> In more detail, we could have an interface; something like this:
> {code}
> public interface OozieActionConfigurator {
> public void updateOozieActionConfiguration(Configuration conf);
> }
> {code}
> that the user can implement, create a jar, and include with their MR action
> (i.e. add a "{{}}" field that let's them specify the class
> name). To protect the Oozie server from running user code (which could do
> anything it wants really), it would have to be run in the Launcher Job. The
> Launcher Job could call this method after it loads the configuration prepared
> by the Oozie server.
> Another thing this will be helpful is with users who use the Java action to
> launch MR jobs and expect a bunch of things to be done for them that are not
> (e.g. delegation token propagation, config loading, returning the hadoop job
> to Oozie, etc). These are all done with the MR action, so the more users we
> can move to the MR action from the Java action, the less they'll run into
> these difficulties.
> Some of this may change slightly as I try to actually implement this (e.g.
> have to handle throwing exceptions etc). And one thing I may do is keep this
> general enough that it should be compatible with all action types in case we
> want to add this to any of them in the future; though for now, the schema
> would only accept it for the MapReduce action.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1954) Add a way for the MapReduce action to be configured by Java code
[
https://issues.apache.org/jira/browse/OOZIE-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14082479#comment-14082479
]
Robert Kanter commented on OOZIE-1954:
--
Test failures unrelated
> Add a way for the MapReduce action to be configured by Java code
>
>
> Key: OOZIE-1954
> URL: https://issues.apache.org/jira/browse/OOZIE-1954
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: OOZIE-1954.patch
>
>
> With certain other components (e.g. Avro, HFileOutputFormat (HBase), etc), it
> becomes impractical to use the MapReduce action and users must instead use
> the Java action. The problem is that these components require a lot of extra
> configuration that is often hidden from the user in Java code (e.g.
> HFileOutputFormat.configureIncrementalLoad(job, table); which can also
> include decision logic, serialization, and other things that we can't do in
> an XML file directly.
> One way to solve this problem is to allow the user to give the MR action some
> Java code that would do this configuration, similar to how we allow the
> {{}} field to specify an external XML file of configuration
> properties.
> In more detail, we could have an interface; something like this:
> {code}
> public interface OozieActionConfigurator {
> public void updateOozieActionConfiguration(Configuration conf);
> }
> {code}
> that the user can implement, create a jar, and include with their MR action
> (i.e. add a "{{}}" field that let's them specify the class
> name). To protect the Oozie server from running user code (which could do
> anything it wants really), it would have to be run in the Launcher Job. The
> Launcher Job could call this method after it loads the configuration prepared
> by the Oozie server.
> Another thing this will be helpful is with users who use the Java action to
> launch MR jobs and expect a bunch of things to be done for them that are not
> (e.g. delegation token propagation, config loading, returning the hadoop job
> to Oozie, etc). These are all done with the MR action, so the more users we
> can move to the MR action from the Java action, the less they'll run into
> these difficulties.
> Some of this may change slightly as I try to actually implement this (e.g.
> have to handle throwing exceptions etc). And one thing I may do is keep this
> general enough that it should be compatible with all action types in case we
> want to add this to any of them in the future; though for now, the schema
> would only accept it for the MapReduce action.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1954) Add a way for the MapReduce action to be configured by Java code
[
https://issues.apache.org/jira/browse/OOZIE-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14081953#comment-14081953
]
Hadoop QA commented on OOZIE-1954:
--
Testing JIRA OOZIE-1954
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch does adds/modifies 4 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1508
.Tests failed: 1
.Tests errors: 1
.The patch failed the following testcases:
. testEngine(org.apache.oozie.command.coord.TestFutureActionsTimeOut)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/1369/
> Add a way for the MapReduce action to be configured by Java code
>
>
> Key: OOZIE-1954
> URL: https://issues.apache.org/jira/browse/OOZIE-1954
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: OOZIE-1954.patch
>
>
> With certain other components (e.g. Avro, HFileOutputFormat (HBase), etc), it
> becomes impractical to use the MapReduce action and users must instead use
> the Java action. The problem is that these components require a lot of extra
> configuration that is often hidden from the user in Java code (e.g.
> HFileOutputFormat.configureIncrementalLoad(job, table); which can also
> include decision logic, serialization, and other things that we can't do in
> an XML file directly.
> One way to solve this problem is to allow the user to give the MR action some
> Java code that would do this configuration, similar to how we allow the
> {{}} field to specify an external XML file of configuration
> properties.
> In more detail, we could have an interface; something like this:
> {code}
> public interface OozieActionConfigurator {
> public void updateOozieActionConfiguration(Configuration conf);
> }
> {code}
> that the user can implement, create a jar, and include with their MR action
> (i.e. add a "{{}}" field that let's them specify the class
> name). To protect the Oozie server from running user code (which could do
> anything it wants really), it would have to be run in the Launcher Job. The
> Launcher Job could call this method after it loads the configuration prepared
> by the Oozie server.
> Another thing this will be helpful is with users who use the Java action to
> launch MR jobs and expect a bunch of things to be done for them that are not
> (e.g. delegation token propagation, config loading, returning the hadoop job
> to Oozie, etc). These are all done with the MR action, so the more users we
> can move to the MR action from the Java action, the less they'll run into
> these difficulties.
> Some of this may change slightly as I try to actually implement this (e.g.
> have to handle throwing exceptions etc). And one thing I may do is keep this
> general enough that it should be compatible with all action types in case we
> want to add this to any of them in the future; though for now, the schema
> would only accept it for the MapReduce action.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
