I think a lot of you are getting bugs because you're trying to build with both the "hadoop-default" / "hadoop2" and the "hadoop3" profile active at the same time, and that's causing dependency resolution issues because the classpath is polluted with a bunch of extra stuff (which you can see by adding in a `dependency:tree` to the maven command).
If you're building using `-Phadoop3`, you should be aware that won't de-activate the default profile, which is set up for Hadoop 2. The correct way to build using the Hadoop 3 profile is with `-Dhadoop.profile=3`, which activates the Hadoop 3 profile and de-activates the default (Hadoop 2) profile at the same time: `mvn clean package -Dtest=CredentialProviderFactoryShimTest -Dhadoop.profile=3` When built correctly, I cannot reproduce the errors seen, but I can reproduce it when I use the incorrect command. Hadoop itself ships with commons-beanutils, and this credential provider code is a Hadoop-specific feature. We don't have a dependency on it for our code and don't ship beanutils in our tarball. This isn't an issue in production, as long as you have your Hadoop libs on your classpath correctly. The reason we use commons-configuration 1.6 is because that converges with what Hadoop uses for [2.6.5,3.0). So, I don't think we should change the version of that. We do ship it in our tarball, because Hadoop 3.0.3 and later use commons-configuration2, so commons-configuration needs to be on the classpath when using Hadoop 3, but we don't want the version to conflict with what Hadoop 2 is using, so we ship the same version. We can expect commons-beanutils to be there, though, since Hadoop ships that in all its releases. We have enough votes that I could close the vote now, but I'll give it another day for anybody who ran into issues to re-test with the correct command, if they want, now that they have the updated information. On Tue, Apr 11, 2023 at 3:21 PM Mark Owens <jmoapa...@gmail.com> wrote: > > If I update the pom's with the commons-configuration changes suggested by > Dave in a previous post, I can successfully run the Sunny profile tests > with both Hadoop 2 and Hadoop 3. > > On Tue, Apr 11, 2023 at 1:16 PM Daniel Roberts <ddani...@gmail.com> wrote: > > > Yes, I changed the memory settings last week and was able to pass the > > ExamplesIT.testScansWithInterference and > > ExamplesIT.testIsolatedScansWithInterference > > without issue. Given these environmental tweaks, do we have a documented > > minimum machine spec that we expect the release tests to pass on? > > > > On Tue, Apr 11, 2023 at 1:00 PM Christopher Shannon < > > christopher.l.shan...@gmail.com> wrote: > > > > > The test failure for ExamplesIT.testScansWithInterference is likely from > > > out of memory. See https://github.com/apache/accumulo/issues/3281 > > > > > > On Tue, Apr 11, 2023 at 12:21 PM Dominic Garguilo < > > > dominic.gargu...@gmail.com> wrote: > > > > > > > After running > > > > mvn clean verify -Phadoop3,sunny > > > > I am consistently seeing the following: > > > > > > > > [ERROR] > > > > > > > > > > > > > > org.apache.accumulo.core.conf.CredentialProviderFactoryShimTest.extractFromHdfs > > > > > Time elapsed: 3.605 s <<< ERROR! > > > > > java.lang.NoClassDefFoundError: > > > > > org/apache/commons/beanutils/BeanIntrospector > > > > > at > > > > > > > > > > > > > > org.apache.accumulo.core.conf.CredentialProviderFactoryShimTest.extractFromHdfs(CredentialProviderFactoryShimTest.java:185) > > > > > Caused by: java.lang.ClassNotFoundException: > > > > > org.apache.commons.beanutils.BeanIntrospector > > > > > at > > > > > > > > > > > > > > org.apache.accumulo.core.conf.CredentialProviderFactoryShimTest.extractFromHdfs(CredentialProviderFactoryShimTest.java:185) > > > > > > > > > > > > This test passes consistently when run in my IDE but fails every time > > > when > > > > running the mentioned maven command. > > > > > > > > On Mon, Apr 10, 2023 at 6:04 PM Dave Marion <dmario...@gmail.com> > > wrote: > > > > > > > > > *mvn clean verify -Phadoop3,sunny* failed in ExamplesIT with the > > > > > aforementioned changes. Specifically, > > > > ExamplesIT.testScansWithInterference > > > > > failed with a server error when closing the batch writer. No further > > > > > information in the logs. > > > > > > > > > > On Mon, Apr 10, 2023 at 5:06 PM Dave Marion <dmario...@gmail.com> > > > wrote: > > > > > > > > > > > Updating commons-configuration from 1.6 to 1.10 did the trick I > > > think. > > > > > > Versions 1.7, 1.8, and 1.9 had compilation issues. I'm not veto-ing > > > > > because > > > > > > other people have built with the hadoop3 profile even though more > > > than > > > > > one > > > > > > of us have had an issue. Here are the changes I made to get me past > > > the > > > > > > CredentialProviderFactoryShimTest test failure in conf using the > > > > hadoop3 > > > > > > profile. > > > > > > > > > > > > > > > > > > diff --git a/core/pom.xml b/core/pom.xml > > > > > > index 163a1c062f..2aad10486a 100644 > > > > > > --- a/core/pom.xml > > > > > > +++ b/core/pom.xml > > > > > > @@ -201,6 +201,8 @@ > > > > > > > > > > > > > > > > > > > > > > > > > > <allow>org[.]apache[.]hadoop[.]mapred[.](FileAlreadyExistsException|InvalidJobConfException)</allow> > > > > > > <!-- allow lexicoders to throw iterator exceptions > > > --> > > > > > > > > > > > > > > > > > > > > > > > > > > <allow>org[.]apache[.]accumulo[.]core[.]iterators[.]ValueFormatException</allow> > > > > > > + <!-- allow commons-configuration for deprecated > > > > > > ClientConfiguration constructor --> > > > > > > + > > > > > > > > > <allow>org[.]apache[.]commons[.]configuration[.]Configuration</allow> > > > > > > </allows> > > > > > > </configuration> > > > > > > </execution> > > > > > > diff --git a/pom.xml b/pom.xml > > > > > > index 854443da2d..2193b46b54 100644 > > > > > > --- a/pom.xml > > > > > > +++ b/pom.xml > > > > > > @@ -212,7 +212,7 @@ > > > > > > <dependency> > > > > > > <groupId>commons-configuration</groupId> > > > > > > <artifactId>commons-configuration</artifactId> > > > > > > - <version>1.6</version> > > > > > > + <version>1.10</version> > > > > > > </dependency> > > > > > > <dependency> > > > > > > <groupId>commons-io</groupId> > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Apr 10, 2023 at 12:16 PM Dave Marion <dmario...@gmail.com> > > > > > wrote: > > > > > > > > > > > >> From what I can tell: > > > > > >> > > > > > >> 1. commons-configuration:1.6 has a commons-beanutils:1.7.0 > > > dependency > > > > > >> 2. hadoop-common:3.0.3 has a commons-beanutils:1.9.3 dependency, > > > > > >> hadoop-common:2.6.5 has no such dependency > > > > > >> > > > > > >> Hadoop 3.0.3 is failing because the commons-beanutils:1.9.3 > > > dependency > > > > > >> must not be on the classpath for the test. I have no idea why this > > > is > > > > > >> failing on my machine and not for others. I'm also seeing general > > > > > slowness > > > > > >> building 1.10.3 on my machine, but not building other git > > branches. > > > > > >> > > > > > >> On Mon, Apr 10, 2023 at 8:24 AM Dave Marion <dmario...@gmail.com> > > > > > wrote: > > > > > >> > > > > > >>> I built the source tarball using the hadoop3 profile again and > > > > > >>> encountered the issue below. The version of hadoop (3.0.3) has > > not > > > > > changed, > > > > > >>> so I assume that this is related to a change in our dependencies. > > > > > >>> > > > > > >>> [ERROR] Tests run: 10, Failures: 0, Errors: 1, Skipped: 0, Time > > > > > elapsed: > > > > > >>> 3.856 s <<< FAILURE! - in > > > > > >>> org.apache.accumulo.core.conf.CredentialProviderFactoryShimTest > > > > > >>> [ERROR] > > > > > >>> > > > > > > > > > > > > > > org.apache.accumulo.core.conf.CredentialProviderFactoryShimTest.extractFromHdfs > > > > > >>> Time elapsed: 1.186 s <<< ERROR! > > > > > >>> java.lang.NoClassDefFoundError: > > > > > >>> org/apache/commons/beanutils/BeanIntrospector > > > > > >>> at > > > > > >>> > > > > > > > > > > > > > > org.apache.accumulo.core.conf.CredentialProviderFactoryShimTest.extractFromHdfs(CredentialProviderFactoryShimTest.java:185) > > > > > >>> Caused by: java.lang.ClassNotFoundException: > > > > > >>> org.apache.commons.beanutils.BeanIntrospector > > > > > >>> at > > > > > >>> > > > > > > > > > > > > > > org.apache.accumulo.core.conf.CredentialProviderFactoryShimTest.extractFromHdfs(CredentialProviderFactoryShimTest.java:185) > > > > > >>> > > > > > >>> On Sun, Apr 9, 2023 at 11:41 AM Jeffrey Manno < > > > > > jeffreymann...@gmail.com> > > > > > >>> wrote: > > > > > >>> > > > > > >>>> +1 > > > > > >>>> > > > > > >>>> I did run into a few flaky tests but none seem to warrant a > > > stoppage > > > > > to > > > > > >>>> this release candidate. Two common ones are SuspendTabletsIT and > > > > > >>>> DeleteTableDuringSplitIT. > > > > > >>>> > > > > > >>>> Other checks I did: > > > > > >>>> > > > > > >>>> - Ran the entire test suite, along with sunny day tests. > > > > > >>>> - Verified checksums. > > > > > >>>> - Did some small ingest testing with agitation. > > > > > >>>> > > > > > >>>> > > > > > >>>> On Sat, Apr 8, 2023 at 9:18 AM Christopher Shannon < > > > > > >>>> christopher.l.shan...@gmail.com> wrote: > > > > > >>>> > > > > > >>>> > +1 (binding), LGTM > > > > > >>>> > > > > > > >>>> > Some of the things I did for verification/testing locally: > > > > > >>>> > > > > > > >>>> > * Validated signatures and checksums > > > > > >>>> > * Verified license and notice files in archives > > > > > >>>> > * Verified source license headers with 'mvn apache-rat:check' > > > > > >>>> > * Built and ran all the sunny integration tests > > > > > >>>> > * Started up using Uno to make sure everything started > > correctly > > > > > >>>> (also ran > > > > > >>>> > a couple scans, etc) > > > > > >>>> > > > > > > >>>> > On Fri, Apr 7, 2023 at 4:23 PM Christopher < > > ctubb...@apache.org > > > > > > > > > >>>> wrote: > > > > > >>>> > > > > > > >>>> > > Accumulo Developers, > > > > > >>>> > > > > > > > >>>> > > Please consider the following candidate for Apache Accumulo > > > > > 1.10.3. > > > > > >>>> > > > > > > > >>>> > > Git Commit: > > > > > >>>> > > 733863638d85d0109d217da7ea5f36f5e483e207 > > > > > >>>> > > Branch: > > > > > >>>> > > 1.10.3-rc1 > > > > > >>>> > > > > > > > >>>> > > If this vote passes, a gpg-signed tag will be created using: > > > > > >>>> > > git tag -f -s -m 'Apache Accumulo 1.10.3' rel/1.10.3 \ > > > > > >>>> > > 733863638d85d0109d217da7ea5f36f5e483e207 > > > > > >>>> > > > > > > > >>>> > > Staging repo: > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > > > > > > > > > > https://repository.apache.org/content/repositories/orgapacheaccumulo-1100 > > > > > >>>> > > Source (official release artifact): > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > > > > > > > > > > https://repository.apache.org/content/repositories/orgapacheaccumulo-1100/org/apache/accumulo/accumulo/1.10.3/accumulo-1.10.3-src.tar.gz > > > > > >>>> > > Binary: > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > > > > > > > > > > https://repository.apache.org/content/repositories/orgapacheaccumulo-1100/org/apache/accumulo/accumulo/1.10.3/accumulo-1.10.3-bin.tar.gz > > > > > >>>> > > > > > > > >>>> > > Append ".asc" to download the cryptographic signature for a > > > > given > > > > > >>>> > artifact. > > > > > >>>> > > (You can also append ".sha1" or ".md5" instead in order to > > > > verify > > > > > >>>> the > > > > > >>>> > > checksums > > > > > >>>> > > generated by Maven to verify the integrity of the Nexus > > > > repository > > > > > >>>> > > staging area.) > > > > > >>>> > > > > > > > >>>> > > Signing keys are available at > > > > > >>>> https://www.apache.org/dist/accumulo/KEYS > > > > > >>>> > > (Expected fingerprint: > > > 8CC4F8A2B29C2B040F2B835D6F0CDAE700B6899D) > > > > > >>>> > > > > > > > >>>> > > In addition to the tarballs and their signatures, the > > > following > > > > > >>>> checksum > > > > > >>>> > > files will be added to the dist/release SVN area after > > > release: > > > > > >>>> > > accumulo-1.10.3-src.tar.gz.sha512 will contain: > > > > > >>>> > > SHA512 (accumulo-1.10.3-src.tar.gz) = > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > > > > > > > > > > 436707da7424ea1b7993355d70747068264276234a4772f9da3866a638ad7860a754436f34ab2108e54801cced67f00ca2cfe0f46dfb80f2d7609b7b7112045e > > > > > >>>> > > accumulo-1.10.3-bin.tar.gz.sha512 will contain: > > > > > >>>> > > SHA512 (accumulo-1.10.3-bin.tar.gz) = > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > > > > > > > > > > 36e6795ad3720ba72fc9f4ddabf45f6d67cdc77658a181733fa2c47bfd3799f123b8840a79b538b8504a7b4bdc97fd0b52efab790395a5e022a897bc18405d0c > > > > > >>>> > > > > > > > >>>> > > Release notes (in progress) can be found at: > > > > > >>>> > > https://accumulo.staged.apache.org/release/accumulo-1.10.3 > > > > > >>>> > > > > > > > >>>> > > Release testing instructions: > > > > > >>>> > > https://accumulo.apache.org/contributor/verifying-release > > > > > >>>> > > > > > > > >>>> > > Please vote one of: > > > > > >>>> > > [ ] +1 - I have verified and accept... > > > > > >>>> > > [ ] +0 - I have reservations, but not strong enough to vote > > > > > >>>> against... > > > > > >>>> > > [ ] -1 - Because..., I do not accept... > > > > > >>>> > > ... these artifacts as the 1.10.3 release of Apache > > Accumulo. > > > > > >>>> > > > > > > > >>>> > > This vote will remain open until at least Mon Apr 10 > > 08:30:00 > > > PM > > > > > UTC > > > > > >>>> > 2023. > > > > > >>>> > > (Mon Apr 10 04:30:00 PM EDT 2023 / Mon Apr 10 01:30:00 PM > > PDT > > > > > 2023) > > > > > >>>> > > Voting can continue after this deadline until the release > > > > manager > > > > > >>>> > > sends an email ending the vote. > > > > > >>>> > > > > > > > >>>> > > Thanks! > > > > > >>>> > > > > > > > >>>> > > P.S. Hint: download the whole staging repo with > > > > > >>>> > > wget -erobots=off -r -l inf -np -nH \ > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > > > > > > > > > > https://repository.apache.org/content/repositories/orgapacheaccumulo-1100/ > > > > > >>>> > > # note the trailing slash is needed > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>> > > > > > > > > > > > > > >
