Hi Karl,
it works perfectly so this means that we don't need to upgrade to a newer
version of Hadoop.
I added the following sed instruction to the ManifoldCF SDK using version
2.26 and it worked:
sed -i -e
's/<\/dependencies>/<dependency><groupId>org.apache.hadoop<\/groupId><artifactId>hadoop-annotations<\/artifactId><version>${hadoop.version}<\/version><exclusions><exclusion><groupId>jdk.tools<\/groupId><artifactId>jdk.tools<\/artifactId><\/exclusion><\/exclusions><\/dependency><\/dependencies>/g'
connectors/hdfs/pom.xml
We have to declare the following dependency in the connectors/hdfs/pom.xml:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-annotations</artifactId>
<version>${hadoop.version}</version>
<exclusions>
<exclusion>
<groupId>jdk.tools</groupId>
<artifactId>jdk.tools</artifactId>
</exclusion>
</exclusions>
</dependency>
Cheers,
PG
Il giorno ven 24 nov 2023 alle ore 23:26 Piergiorgio Lucidi <
piergior...@apache.org> ha scritto:
> Hi Karl,
>
> the problem is that the unique dependency declared in that pom.xml is the
> hadoop-common importing hadoop-annotations that is causing the issue.
> Below what I see in the hadoop-annotations pom:
>
> <profiles>
> <profile>
> <id>os.linux</id>
> <activation>
> <os>
> <family>!Mac</family>
> </os>
> </activation>
> <dependencies>
> <dependency>
> <groupId>jdk.tools</groupId>
> <artifactId>jdk.tools</artifactId>
> <version>1.6</version>
> <scope>system</scope>
> <systemPath>${java.home}/../lib/tools.jar</systemPath>
> </dependency>
> </dependencies>
> </profile>
> <profile>
> <id>jdk1.7</id>
> <activation>
> <jdk>1.7</jdk>
> </activation>
> <dependencies>
> <dependency>
> <groupId>jdk.tools</groupId>
> <artifactId>jdk.tools</artifactId>
> <version>1.7</version>
> <scope>system</scope>
> <systemPath>${java.home}/../lib/tools.jar</systemPath>
> </dependency>
> </dependencies>
> </profile>
> </profiles>
>
> Probably we could solve declaring hadoop-annotations dependency by adding
> an exclusion related to the jdk.tools dependency.
> Something like this:
>
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-annotations</artifactId>
> <version>${hadoop.version}</version>
> <exclusions>
> <exclusion>
> <groupId>jdk.tools</groupId>
> <artifactId>jdk.tools</artifactId>
> </exclusion>
> </exclusions>
> </dependency>
>
> I'll try to apply this using the ManifoldCF SDK and I'll let you know
>
> Cheers,
> PG
>
>
> Il giorno ven 24 nov 2023 alle ore 21:34 Karl Wright <daddy...@gmail.com>
> ha scritto:
>
>> Hi - the jar it's looking for may no longer be part of the java 11 jdk.
>> I'm not exactly sure how to best handle this in Maven. It may simply be
>> possible to remove the dependency entirely from the maven pom.
>>
>>
>> On Fri, Nov 24, 2023 at 12:34 PM Piergiorgio Lucidi <
>> piergior...@apache.org>
>> wrote:
>>
>> > I have just created this ticket:
>> > https://issues.apache.org/jira/browse/CONNECTORS-1751
>> >
>> > Il giorno ven 24 nov 2023 alle ore 18:19 Piergiorgio Lucidi <
>> > piergior...@apache.org> ha scritto:
>> >
>> > > Hi folks,
>> > >
>> > > Thanks to the ManifoldCF SDK, I found another issue with the Maven
>> build.
>> > > Steps to reproduce the issue with the Manifold SDK:
>> > >
>> > > 1. Clone the ManifoldCF SDK project from the following URL:
>> > > git clone https://github.com/OpenPj/manifoldcf-sdk.git
>> > >
>> > > 2. Remove row 27 from the run.sh script, this line is including the
>> > hotfix
>> > > for the hadoop version upgrading it to the latest version 3.3.6.
>> > >
>> > > 3. Run the following command in order to download the ManifoldCF
>> source
>> > > code and run the Ant and Maven build process in a Docker container
>> based
>> > on
>> > > Maven 3.9.5 and OpenJDK Temurin 11This in order to install locally all
>> > the
>> > > Maven dependencies needed to implement custom extensions / connectors.
>> > The
>> > > Docker Volume includes the entire Maven repo used by the ManifoldCF
>> build
>> > > process and it will be copied in the SDK target folder and configured
>> in
>> > > order to compile your Java Custom code:
>> > >
>> > > ./run.sh init 2.26 ga
>> > >
>> > > The current version of the SDK includes the sep instruction to fix all
>> > the
>> > > build problems described here:
>> > > https://issues.apache.org/jira/browse/CONNECTORS-1750
>> > >
>> > > These issues are also resolved in ManifoldCF main trunk but still
>> present
>> > > in the latest previous release packages (source code packages).
>> > > I have to confess that I don't know if this issue is also included in
>> > > other releases.
>> > > The SDK is returning the following error that should be related to the
>> > > Hadoop Annotations dependencies of Hadoop 2.6.0, a very old version of
>> > > Hadoop that includes a JDK 1.6 dependency:
>> jdk.tools:jdk.tools:jar:1.6.
>> > >
>> > > [ERROR] Failed to execute goal on project mcf-hdfs-connector: Could
>> not
>> > > resolve dependencies for project
>> > > org.apache.manifoldcf:mcf-hdfs-connector:jar:2.26: The following
>> > artifacts
>> > > could not be resolved: jdk.tools:jdk.tools:jar:1.6: Could not find
>> > artifact
>> > > jdk.tools:jdk.tools:jar:1.6 at specified path
>> > > /opt/java/openjdk/../lib/tools.jar -> [Help 1]
>> > > org.apache.maven.lifecycle.LifecycleExecutionException: Failed to
>> execute
>> > > goal on project mcf-hdfs-connector: Could not resolve dependencies for
>> > > project org.apache.manifoldcf:mcf-hdfs-connector:jar:2.26: The
>> following
>> > > artifacts could not be resolved: jdk.tools:jdk.tools:jar:1.6: Could
>> not
>> > > find artifact jdk.tools:jdk.tools:jar:1.6 at specified path
>> > > /opt/java/openjdk/../lib/tools.jar
>> > > at
>> > >
>> >
>> org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies
>> > > (LifecycleDependencyResolver.java:243)
>> > > at
>> > >
>> >
>> org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies
>> > > (LifecycleDependencyResolver.java:136)
>> > > at
>> > >
>> >
>> org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved
>> > > (MojoExecutor.java:355)
>> > > at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute
>> > > (MojoExecutor.java:313)
>> > > at org.apache.maven.lifecycle.internal.MojoExecutor.execute
>> > > (MojoExecutor.java:212)
>> > > at org.apache.maven.lifecycle.internal.MojoExecutor.execute
>> > > (MojoExecutor.java:174)
>> > > at org.apache.maven.lifecycle.internal.MojoExecutor.access$000
>> > > (MojoExecutor.java:75)
>> > > at org.apache.maven.lifecycle.internal.MojoExecutor$1.run
>> > > (MojoExecutor.java:162)
>> > > at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute
>> > > (DefaultMojosExecutionStrategy.java:39)
>> > > at org.apache.maven.lifecycle.internal.MojoExecutor.execute
>> > > (MojoExecutor.java:159)
>> > > at
>> > >
>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
>> > > (LifecycleModuleBuilder.java:105)
>> > > at
>> > >
>> >
>> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call
>> > > (MultiThreadedBuilder.java:193)
>> > > at
>> > >
>> >
>> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call
>> > > (MultiThreadedBuilder.java:180)
>> > > at java.util.concurrent.FutureTask.run (FutureTask.java:264)
>> > > at java.util.concurrent.Executors$RunnableAdapter.call
>> > > (Executors.java:515)
>> > > at java.util.concurrent.FutureTask.run (FutureTask.java:264)
>> > > at java.util.concurrent.ThreadPoolExecutor.runWorker
>> > > (ThreadPoolExecutor.java:1128)
>> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run
>> > > (ThreadPoolExecutor.java:628)
>> > > at java.lang.Thread.run (Thread.java:829)
>> > > . . .
>> > > Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException:
>> Could
>> > > not find artifact jdk.tools:jdk.tools:jar:1.6 at specified path
>> > > /opt/java/openjdk/../lib/tools.jar
>> > > at
>> org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve
>> > > (DefaultArtifactResolver.java:305)
>> > > at
>> > >
>> org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts
>> > > (DefaultArtifactResolver.java:261)
>> > > at
>> > >
>> >
>> org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies
>> > > (DefaultRepositorySystem.java:353)
>> > > at
>> > org.apache.maven.project.DefaultProjectDependenciesResolver.resolve
>> > > (DefaultProjectDependenciesResolver.java:182)
>> > > at
>> > >
>> >
>> org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies
>> > > (LifecycleDependencyResolver.java:224)
>> > > at
>> > >
>> >
>> org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies
>> > > (LifecycleDependencyResolver.java:136)
>> > > at
>> > >
>> >
>> org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved
>> > > (MojoExecutor.java:355)
>> > > at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute
>> > > (MojoExecutor.java:313)
>> > > at org.apache.maven.lifecycle.internal.MojoExecutor.execute
>> > > (MojoExecutor.java:212)
>> > > at org.apache.maven.lifecycle.internal.MojoExecutor.execute
>> > > (MojoExecutor.java:174)
>> > > at org.apache.maven.lifecycle.internal.MojoExecutor.access$000
>> > > (MojoExecutor.java:75)
>> > > at org.apache.maven.lifecycle.internal.MojoExecutor$1.run
>> > > (MojoExecutor.java:162)
>> > > at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute
>> > > (DefaultMojosExecutionStrategy.java:39)
>> > > at org.apache.maven.lifecycle.internal.MojoExecutor.execute
>> > > (MojoExecutor.java:159)
>> > > at
>> > >
>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
>> > > (LifecycleModuleBuilder.java:105)
>> > > at
>> > >
>> >
>> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call
>> > > (MultiThreadedBuilder.java:193)
>> > > at
>> > >
>> >
>> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call
>> > > (MultiThreadedBuilder.java:180)
>> > > at java.util.concurrent.FutureTask.run (FutureTask.java:264)
>> > > at java.util.concurrent.Executors$RunnableAdapter.call
>> > > (Executors.java:515)
>> > > at java.util.concurrent.FutureTask.run (FutureTask.java:264)
>> > > at java.util.concurrent.ThreadPoolExecutor.runWorker
>> > > (ThreadPoolExecutor.java:1128)
>> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run
>> > > (ThreadPoolExecutor.java:628)
>> > > at java.lang.Thread.run (Thread.java:829)
>> > >
>> > > Trying to use the latest version of Hadoop 3.3.6, reverting to the
>> > > original run.sh file, it solves this issue.
>> > > I would like to use the updated version because even if we found a
>> way to
>> > > exclude this old dependency it sounds strange to use very old
>> libraries
>> > > requiring jdk 1.6.
>> > > I'll create a specific ticket for this in order to discuss if it is
>> > > possible to update to the latest version without any issues.
>> > >
>> > > Please let me know what you think.
>> > > Thanks.
>> > >
>> > > Cheers,
>> > > PG
>> > >
>> > > --
>> > > Piergiorgio
>> > >
>> >
>> >
>> > --
>> > Piergiorgio
>> >
>>
>
>
> --
> Piergiorgio
>
--
Piergiorgio
