[
https://issues.apache.org/jira/browse/HADOOP-11929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14567413#comment-14567413
]
Allen Wittenauer commented on HADOOP-11929:
-------------------------------------------
After a lot of playing/debugging, I've learned that we have a bit of a
dependency race.
As expected, running mvn install will fail for any modules that don't have
their dependencies in the maven cache. Now here's the rub: if the Apache
snapshot repo is out of date (like right now for some Hadoop subprojects), mvn
install can't download those bits.
Up until last night, I was only able to reproduce this locally and in
particular with hadoop-tools. But at about 11pm last night I *was* able to
break it last night during a test run of this patch combo'd with HADOOP-11984.
I'm going to try again today (assuming the repos are still out of date) to
reproduce this on the Jenkins servers.
It's unclear to me what impact this has on unit tests.
What makes this particular hard to see are two things:
* This is a race due to the fact that there is a good chance the 2xdaily builds
keep the local maven repos on the Jenkins boxes fairly fresh.
* An extremely high amount of patches don't hit low activity areas (such as
hadoop-tools) so this race is never exposed.
I'm thinking about combating this in one or two ways:
* Run a mvn install of the patched modules as part of pre-patch. This will
help explain failed mvn install in post-patch. This is fairly cheap, but
doesn't really fix the problem.
* Run a full mvn install of the entire repo after the javac test. This will
keep the local repo as up to date as possible for all dependencies, even out of
the way ones. This is fairly expensive.
Thoughts?
I'll likely be posting another combo patch later today. I have some issues in
checkstyle to work out first though. I can tell I'm getting very close to
having something finished here.
> add test-patch plugin points for customizing build layout
> ---------------------------------------------------------
>
> Key: HADOOP-11929
> URL: https://issues.apache.org/jira/browse/HADOOP-11929
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Sean Busbey
> Assignee: Allen Wittenauer
> Priority: Minor
> Attachments: HADOOP-11929.00.patch, HADOOP-11929.01.patch,
> HADOOP-11929.02.patch, HADOOP-11929.03.patch, hadoop.sh
>
>
> Sean Busbey and I had a chat about this at the Bug Bash. Here's the proposal:
> * Introduce the concept of a 'personality module'.
> * There can be only one personality.
> * Personalities provide a single function that takes as input the name of
> the test current being processed
> * This function uses two other built-in functions to define two queues:
> maven module name and profiles to use against those maven module names
> * If something needs to be compiled prior to this test (but not actually
> tested), the personality will be responsible for doing that compilation
> In hadoop, the classic example is hadoop-hdfs needs common compiled with the
> native bits. So prior to the javac tests, the personality would check
> CHANGED_MODULES, see hadoop-hdfs, and compile common w/ -Pnative prior to
> letting test-patch.sh do the work in hadoop-hdfs. Another example is our lack
> of test coverage of various native bits. Since these require profiles to be
> defined prior to compilation, the personality could see that something
> touches native code, set the appropriate profile, and let test-patch.sh be on
> its way.
> One way to think of it is some higher order logic on top of the automated
> 'figure out what modules and what tests to run' functions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
