> On Aug 25, 2017, at 10:00 AM, Steve Loughran <[email protected]> wrote:
>
> Catching up on this. Looks like I don't have a hadoop-aws profile, which
> explains a lot, doesn't it.
Yes. This is exactly the type of failure I'd expect.
> How do those profiles get created/copied in?
Maven kludgery.
In a hadoop-tools sub-module pom.xml, you'll find an entry like this or
similar:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>deplist</id>
<phase>compile</phase>
<goals>
<goal>list</goal>
</goals>
<configuration>
<!-- build a shellprofile -->
<outputFile>${project.basedir}/target/hadoop-tools-deps/${project.artifactId}.tools-optional.txt</outputFile>
</configuration>
</execution>
</executions>
</plugin>
The files generated by this entry get read by
dev-support/bin/dist-tools-hooks-maker. That script is run as part of -Pdist
in hadoop-dist. The outputFile name determines what kind of support hook it
makes. (There were a lot of bad decisions made in nomenclature here. I take
full responsibility for the confusion. But it makes more sense when one views
the names from the perspective of the code in hadoop-functions.sh)
All/most of this hackery should probably get replaced by something
smarter in the hadoop-maven-plugin. But for the most part, this does work
though and makes the end user experience significantly better.
> I know there's an explicit s3guard entry now.
>
> hadoop-tools/hadoop-aws/src/main/shellprofile.d/hadoop-s3guard.sh
>
> ..do you think the presence of that entry is causing problems (i.e stopping a
> hadoop-aws profile being created?)
I can confirm that HADOOP-13345 doesn't get a
shellprofile.d/hadoop-aws.sh created. That's not good. I don't have time right
now to dig deep, but a few things pop into my head:
* multiple org.apache.maven.plugins definitions in the pom.xml (do all of them
get executed or just the last one?)
* dist-tools-hooks-maker may only allowed one of builtin or optional . may need
to define a 3rd type that does a smart version of both
* -Pdist may only allow one shellprofile.d dir per module ?
If you want, file a jira and assign it to me. I'll try and dig into it
next week.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
