[
https://issues.apache.org/jira/browse/OOZIE-2714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16259420#comment-16259420
]
Hadoop QA commented on OOZIE-2714:
----------------------------------
Testing JIRA OOZIE-2714
Cleaning local git workspace
----------------------------
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
. {color:green}+1{color} the patch does not introduce any @author tags
. {color:green}+1{color} the patch does not introduce any tabs
. {color:green}+1{color} the patch does not introduce any trailing spaces
. {color:green}+1{color} the patch does not introduce any line longer than
132
. {color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
. {color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
. {color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
. {color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
. {color:green}+1{color} HEAD compiles
. {color:green}+1{color} patch compiles
. {color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1{color} There are [2] new bugs found below threshold in total that
must be fixed.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:red}-1{color} There are [2] new bugs found below threshold in
[sharelib/oozie] that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones):
sharelib/oozie/findbugs-new.html
. The most important FindBugs errors are:
. At ConflictDetectingClassLoader.java:[line 106]: File(...) reads a file whose
location might be specified by user input
. Exception is caught when Exception is not thrown in
org.apache.oozie.action.hadoop.ConflictDetectingClassLoader.loadClass(String):
At ConflictDetectingClassLoader.java:[line 106]
. At ConflictDetectingClassLoader.java:[line 114]
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [server].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
. {color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
. {color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
. Tests run: 1860
. Tests failed: 36
. Tests errors: 0
. The patch failed the following testcases:
. testHiveAction(org.apache.oozie.action.hadoop.TestHiveActionExecutor)
. testSparkAction(org.apache.oozie.action.hadoop.TestSparkActionExecutor)
. testPyspark(org.apache.oozie.action.hadoop.TestPyspark)
. testHive2Action(org.apache.oozie.action.hadoop.TestHive2ActionExecutor)
.
testHive2ActionFails(org.apache.oozie.action.hadoop.TestHive2ActionExecutor)
.
testMapReduceWithUberJarEnabled(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
.
testSetMapredJobName(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
.
testEndWithoutConfiguration(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
.
testMapReduceWithConfigClass(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
.
testMapReduceWithCredentials(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
.
testJobNameSetForMapReduceChild(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
.
testStreamingConfOverride(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
.
testMapReduceWithConfigClassThrowException(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
. testStreaming(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
.
testSetExecutionStats_when_user_has_specified_stats_write_TRUE(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
.
testSetExecutionStats_when_user_has_specified_stats_write_FALSE(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
.
testMapReduceWithConfigClassNotFound(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
. testMapReduce(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
.
testMapReduceActionError(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
.
testMapReduceActionKill(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
. testPig(org.apache.oozie.action.hadoop.TestPigActionExecutor)
.
testExecutionStatsWithMaxStatsSizeLimit(org.apache.oozie.action.hadoop.TestPigActionExecutor)
. testPigError(org.apache.oozie.action.hadoop.TestPigActionExecutor)
.
testExecutionStatsWithRetrieveStatsFalse(org.apache.oozie.action.hadoop.TestPigActionExecutor)
.
testExternalChildIds(org.apache.oozie.action.hadoop.TestPigActionExecutor)
. testExecutionStats(org.apache.oozie.action.hadoop.TestPigActionExecutor)
. testUdfPig(org.apache.oozie.action.hadoop.TestPigActionExecutor)
.
testSqoopActionWithRedundantArgsAndFreeFormQuery(org.apache.oozie.action.hadoop.TestSqoopActionExecutor)
.
testSqoopActionWithArgsAndFreeFormQuery(org.apache.oozie.action.hadoop.TestSqoopActionExecutor)
.
testSqoopActionWithBadRedundantArgsAndFreeFormQuery(org.apache.oozie.action.hadoop.TestSqoopActionExecutor)
.
testSqoopActionWithBadCommand(org.apache.oozie.action.hadoop.TestSqoopActionExecutor)
. testSqoopEval(org.apache.oozie.action.hadoop.TestSqoopActionExecutor)
.
testSqoopActionWithRedundantPrefix(org.apache.oozie.action.hadoop.TestSqoopActionExecutor)
. testSqoopAction(org.apache.oozie.action.hadoop.TestSqoopActionExecutor)
.
testParallelFsAndShellWorkflowCompletesSuccessfully(org.apache.oozie.test.TestWorkflowRetries)
.
testParallelFsAndShellWorkflowCompletesSuccessfully(org.apache.oozie.test.TestWorkflow)
. Tests failing with errors:
.
{color:green}+1 DISTRO{color}
. {color:green}+1{color} distro tarball builds with the patch
----------------------------
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
{color:red}. There is at least one warning, please check{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/245/
> Detect conflicting resources during class loading
> -------------------------------------------------
>
> Key: OOZIE-2714
> URL: https://issues.apache.org/jira/browse/OOZIE-2714
> Project: Oozie
> Issue Type: New Feature
> Components: core
> Reporter: Peter Bacsko
> Assignee: Peter Bacsko
> Attachments: ClassLoaderTest.java, OOZIE-2714-POC01.patch,
> OOZIE-2714-POC02.patch
>
>
> There are a bunch of issues in Oozie which are related to class loading.
> The main problem is that the classpath is constructed in a way which is very
> specific to Oozie:
> - Hadoop lib jars
> - Sharelib jars
> - User-defined jars
> Sometimes there is a conflict between sharelib and hadoop lib version. Also,
> users can add their own jars which sometimes contain a different version of
> popular libraries such as Guava, Apache commons, etc.
> We should be able to detect these conflicts and print exact error message so
> that Oozie users can take appropriate actions to resolve the problem.
> A possible approach is the following:
> * start the execution of an action on a different thread
> * replace the thread's context classloader with a classloader which can
> detect conflicts
> * when the JVM invokes the {{loadClass()}} method of the classloader, it
> scans through the jars (which are available as {{URLClassPath}} objects). If
> it finds the given resource in at least two jars, it can do different things
> depending on the setup:
> ** throws an error immediately, mentioning the conflicting jars (this is
> probably too strict - but still an option)
> ** loads the two resource into a byte array and compares them - it only
> throws an error if there is difference
> ** compares the jars but only emits an error message if there is a conflict
> ** something else (user defined action?)
> Implementing such a classloader is not difficult and would greatly enhance
> the supportability of Oozie. It could work in multiple modes depending on the
> setup - perhaps being able to control it from a workflow config is desirable.
> If there's any problem, we should be able to turn it off completely, too.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
