Thanks Christopher for setting me straight. This information did enable the build to move beyond the CredentialProviderFactoryShimTest. I'm unable to get past MiniAccumuloClusterTest though. Two tests timeout for me every time even when increasing the timeout factor (they both specify a timeout in the Test annotation). I was able to verify hashes and keys, but I'm going to refrain from voting as I was really unable to test anything because of my various issues. I think we should move forward though based on others' testing and votes.
On Tue, Apr 11, 2023 at 7:01 PM Christopher <ctubb...@apache.org> wrote: > I think a lot of you are getting bugs because you're trying to build > with both the "hadoop-default" / "hadoop2" and the "hadoop3" profile > active at the same time, and that's causing dependency resolution > issues because the classpath is polluted with a bunch of extra stuff > (which you can see by adding in a `dependency:tree` to the maven > command). > > If you're building using `-Phadoop3`, you should be aware that won't > de-activate the default profile, which is set up for Hadoop 2. The > correct way to build using the Hadoop 3 profile is with > `-Dhadoop.profile=3`, which activates the Hadoop 3 profile and > de-activates the default (Hadoop 2) profile at the same time: > > `mvn clean package -Dtest=CredentialProviderFactoryShimTest > -Dhadoop.profile=3` > > When built correctly, I cannot reproduce the errors seen, but I can > reproduce it when I use the incorrect command. > > Hadoop itself ships with commons-beanutils, and this credential > provider code is a Hadoop-specific feature. We don't have a dependency > on it for our code and don't ship beanutils in our tarball. This isn't > an issue in production, as long as you have your Hadoop libs on your > classpath correctly. The reason we use commons-configuration 1.6 is > because that converges with what Hadoop uses for [2.6.5,3.0). So, I > don't think we should change the version of that. We do ship it in our > tarball, because Hadoop 3.0.3 and later use commons-configuration2, so > commons-configuration needs to be on the classpath when using Hadoop > 3, but we don't want the version to conflict with what Hadoop 2 is > using, so we ship the same version. We can expect commons-beanutils to > be there, though, since Hadoop ships that in all its releases. > > We have enough votes that I could close the vote now, but I'll give it > another day for anybody who ran into issues to re-test with the > correct command, if they want, now that they have the updated > information. > > On Tue, Apr 11, 2023 at 3:21 PM Mark Owens <jmoapa...@gmail.com> wrote: > > > > If I update the pom's with the commons-configuration changes suggested by > > Dave in a previous post, I can successfully run the Sunny profile tests > > with both Hadoop 2 and Hadoop 3. > > > > On Tue, Apr 11, 2023 at 1:16 PM Daniel Roberts <ddani...@gmail.com> > wrote: > > > > > Yes, I changed the memory settings last week and was able to pass the > > > ExamplesIT.testScansWithInterference and > > > ExamplesIT.testIsolatedScansWithInterference > > > without issue. Given these environmental tweaks, do we have a > documented > > > minimum machine spec that we expect the release tests to pass on? > > > > > > On Tue, Apr 11, 2023 at 1:00 PM Christopher Shannon < > > > christopher.l.shan...@gmail.com> wrote: > > > > > > > The test failure for ExamplesIT.testScansWithInterference is likely > from > > > > out of memory. See https://github.com/apache/accumulo/issues/3281 > > > > > > > > On Tue, Apr 11, 2023 at 12:21 PM Dominic Garguilo < > > > > dominic.gargu...@gmail.com> wrote: > > > > > > > > > After running > > > > > mvn clean verify -Phadoop3,sunny > > > > > I am consistently seeing the following: > > > > > > > > > > [ERROR] > > > > > > > > > > > > > > > > > > > org.apache.accumulo.core.conf.CredentialProviderFactoryShimTest.extractFromHdfs > > > > > > Time elapsed: 3.605 s <<< ERROR! > > > > > > java.lang.NoClassDefFoundError: > > > > > > org/apache/commons/beanutils/BeanIntrospector > > > > > > at > > > > > > > > > > > > > > > > > > > org.apache.accumulo.core.conf.CredentialProviderFactoryShimTest.extractFromHdfs(CredentialProviderFactoryShimTest.java:185) > > > > > > Caused by: java.lang.ClassNotFoundException: > > > > > > org.apache.commons.beanutils.BeanIntrospector > > > > > > at > > > > > > > > > > > > > > > > > > > org.apache.accumulo.core.conf.CredentialProviderFactoryShimTest.extractFromHdfs(CredentialProviderFactoryShimTest.java:185) > > > > > > > > > > > > > > > This test passes consistently when run in my IDE but fails every > time > > > > when > > > > > running the mentioned maven command. > > > > > > > > > > On Mon, Apr 10, 2023 at 6:04 PM Dave Marion <dmario...@gmail.com> > > > wrote: > > > > > > > > > > > *mvn clean verify -Phadoop3,sunny* failed in ExamplesIT with the > > > > > > aforementioned changes. Specifically, > > > > > ExamplesIT.testScansWithInterference > > > > > > failed with a server error when closing the batch writer. No > further > > > > > > information in the logs. > > > > > > > > > > > > On Mon, Apr 10, 2023 at 5:06 PM Dave Marion <dmario...@gmail.com > > > > > > wrote: > > > > > > > > > > > > > Updating commons-configuration from 1.6 to 1.10 did the trick I > > > > think. > > > > > > > Versions 1.7, 1.8, and 1.9 had compilation issues. I'm not > veto-ing > > > > > > because > > > > > > > other people have built with the hadoop3 profile even though > more > > > > than > > > > > > one > > > > > > > of us have had an issue. Here are the changes I made to get me > past > > > > the > > > > > > > CredentialProviderFactoryShimTest test failure in conf using > the > > > > > hadoop3 > > > > > > > profile. > > > > > > > > > > > > > > > > > > > > > diff --git a/core/pom.xml b/core/pom.xml > > > > > > > index 163a1c062f..2aad10486a 100644 > > > > > > > --- a/core/pom.xml > > > > > > > +++ b/core/pom.xml > > > > > > > @@ -201,6 +201,8 @@ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > <allow>org[.]apache[.]hadoop[.]mapred[.](FileAlreadyExistsException|InvalidJobConfException)</allow> > > > > > > > <!-- allow lexicoders to throw iterator > exceptions > > > > --> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > <allow>org[.]apache[.]accumulo[.]core[.]iterators[.]ValueFormatException</allow> > > > > > > > + <!-- allow commons-configuration for > deprecated > > > > > > > ClientConfiguration constructor --> > > > > > > > + > > > > > > > > > > > <allow>org[.]apache[.]commons[.]configuration[.]Configuration</allow> > > > > > > > </allows> > > > > > > > </configuration> > > > > > > > </execution> > > > > > > > diff --git a/pom.xml b/pom.xml > > > > > > > index 854443da2d..2193b46b54 100644 > > > > > > > --- a/pom.xml > > > > > > > +++ b/pom.xml > > > > > > > @@ -212,7 +212,7 @@ > > > > > > > <dependency> > > > > > > > <groupId>commons-configuration</groupId> > > > > > > > <artifactId>commons-configuration</artifactId> > > > > > > > - <version>1.6</version> > > > > > > > + <version>1.10</version> > > > > > > > </dependency> > > > > > > > <dependency> > > > > > > > <groupId>commons-io</groupId> > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Apr 10, 2023 at 12:16 PM Dave Marion < > dmario...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > >> From what I can tell: > > > > > > >> > > > > > > >> 1. commons-configuration:1.6 has a commons-beanutils:1.7.0 > > > > dependency > > > > > > >> 2. hadoop-common:3.0.3 has a commons-beanutils:1.9.3 > dependency, > > > > > > >> hadoop-common:2.6.5 has no such dependency > > > > > > >> > > > > > > >> Hadoop 3.0.3 is failing because the commons-beanutils:1.9.3 > > > > dependency > > > > > > >> must not be on the classpath for the test. I have no idea why > this > > > > is > > > > > > >> failing on my machine and not for others. I'm also seeing > general > > > > > > slowness > > > > > > >> building 1.10.3 on my machine, but not building other git > > > branches. > > > > > > >> > > > > > > >> On Mon, Apr 10, 2023 at 8:24 AM Dave Marion < > dmario...@gmail.com> > > > > > > wrote: > > > > > > >> > > > > > > >>> I built the source tarball using the hadoop3 profile again > and > > > > > > >>> encountered the issue below. The version of hadoop (3.0.3) > has > > > not > > > > > > changed, > > > > > > >>> so I assume that this is related to a change in our > dependencies. > > > > > > >>> > > > > > > >>> [ERROR] Tests run: 10, Failures: 0, Errors: 1, Skipped: 0, > Time > > > > > > elapsed: > > > > > > >>> 3.856 s <<< FAILURE! - in > > > > > > >>> > org.apache.accumulo.core.conf.CredentialProviderFactoryShimTest > > > > > > >>> [ERROR] > > > > > > >>> > > > > > > > > > > > > > > > > > > > org.apache.accumulo.core.conf.CredentialProviderFactoryShimTest.extractFromHdfs > > > > > > >>> Time elapsed: 1.186 s <<< ERROR! > > > > > > >>> java.lang.NoClassDefFoundError: > > > > > > >>> org/apache/commons/beanutils/BeanIntrospector > > > > > > >>> at > > > > > > >>> > > > > > > > > > > > > > > > > > > > org.apache.accumulo.core.conf.CredentialProviderFactoryShimTest.extractFromHdfs(CredentialProviderFactoryShimTest.java:185) > > > > > > >>> Caused by: java.lang.ClassNotFoundException: > > > > > > >>> org.apache.commons.beanutils.BeanIntrospector > > > > > > >>> at > > > > > > >>> > > > > > > > > > > > > > > > > > > > org.apache.accumulo.core.conf.CredentialProviderFactoryShimTest.extractFromHdfs(CredentialProviderFactoryShimTest.java:185) > > > > > > >>> > > > > > > >>> On Sun, Apr 9, 2023 at 11:41 AM Jeffrey Manno < > > > > > > jeffreymann...@gmail.com> > > > > > > >>> wrote: > > > > > > >>> > > > > > > >>>> +1 > > > > > > >>>> > > > > > > >>>> I did run into a few flaky tests but none seem to warrant a > > > > stoppage > > > > > > to > > > > > > >>>> this release candidate. Two common ones are > SuspendTabletsIT and > > > > > > >>>> DeleteTableDuringSplitIT. > > > > > > >>>> > > > > > > >>>> Other checks I did: > > > > > > >>>> > > > > > > >>>> - Ran the entire test suite, along with sunny day tests. > > > > > > >>>> - Verified checksums. > > > > > > >>>> - Did some small ingest testing with agitation. > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> On Sat, Apr 8, 2023 at 9:18 AM Christopher Shannon < > > > > > > >>>> christopher.l.shan...@gmail.com> wrote: > > > > > > >>>> > > > > > > >>>> > +1 (binding), LGTM > > > > > > >>>> > > > > > > > >>>> > Some of the things I did for verification/testing locally: > > > > > > >>>> > > > > > > > >>>> > * Validated signatures and checksums > > > > > > >>>> > * Verified license and notice files in archives > > > > > > >>>> > * Verified source license headers with 'mvn > apache-rat:check' > > > > > > >>>> > * Built and ran all the sunny integration tests > > > > > > >>>> > * Started up using Uno to make sure everything started > > > correctly > > > > > > >>>> (also ran > > > > > > >>>> > a couple scans, etc) > > > > > > >>>> > > > > > > > >>>> > On Fri, Apr 7, 2023 at 4:23 PM Christopher < > > > ctubb...@apache.org > > > > > > > > > > > >>>> wrote: > > > > > > >>>> > > > > > > > >>>> > > Accumulo Developers, > > > > > > >>>> > > > > > > > > >>>> > > Please consider the following candidate for Apache > Accumulo > > > > > > 1.10.3. > > > > > > >>>> > > > > > > > > >>>> > > Git Commit: > > > > > > >>>> > > 733863638d85d0109d217da7ea5f36f5e483e207 > > > > > > >>>> > > Branch: > > > > > > >>>> > > 1.10.3-rc1 > > > > > > >>>> > > > > > > > > >>>> > > If this vote passes, a gpg-signed tag will be created > using: > > > > > > >>>> > > git tag -f -s -m 'Apache Accumulo 1.10.3' > rel/1.10.3 \ > > > > > > >>>> > > 733863638d85d0109d217da7ea5f36f5e483e207 > > > > > > >>>> > > > > > > > > >>>> > > Staging repo: > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > https://repository.apache.org/content/repositories/orgapacheaccumulo-1100 > > > > > > >>>> > > Source (official release artifact): > > > > > > >>>> > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > https://repository.apache.org/content/repositories/orgapacheaccumulo-1100/org/apache/accumulo/accumulo/1.10.3/accumulo-1.10.3-src.tar.gz > > > > > > >>>> > > Binary: > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > https://repository.apache.org/content/repositories/orgapacheaccumulo-1100/org/apache/accumulo/accumulo/1.10.3/accumulo-1.10.3-bin.tar.gz > > > > > > >>>> > > > > > > > > >>>> > > Append ".asc" to download the cryptographic signature > for a > > > > > given > > > > > > >>>> > artifact. > > > > > > >>>> > > (You can also append ".sha1" or ".md5" instead in order > to > > > > > verify > > > > > > >>>> the > > > > > > >>>> > > checksums > > > > > > >>>> > > generated by Maven to verify the integrity of the Nexus > > > > > repository > > > > > > >>>> > > staging area.) > > > > > > >>>> > > > > > > > > >>>> > > Signing keys are available at > > > > > > >>>> https://www.apache.org/dist/accumulo/KEYS > > > > > > >>>> > > (Expected fingerprint: > > > > 8CC4F8A2B29C2B040F2B835D6F0CDAE700B6899D) > > > > > > >>>> > > > > > > > > >>>> > > In addition to the tarballs and their signatures, the > > > > following > > > > > > >>>> checksum > > > > > > >>>> > > files will be added to the dist/release SVN area after > > > > release: > > > > > > >>>> > > accumulo-1.10.3-src.tar.gz.sha512 will contain: > > > > > > >>>> > > SHA512 (accumulo-1.10.3-src.tar.gz) = > > > > > > >>>> > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > 436707da7424ea1b7993355d70747068264276234a4772f9da3866a638ad7860a754436f34ab2108e54801cced67f00ca2cfe0f46dfb80f2d7609b7b7112045e > > > > > > >>>> > > accumulo-1.10.3-bin.tar.gz.sha512 will contain: > > > > > > >>>> > > SHA512 (accumulo-1.10.3-bin.tar.gz) = > > > > > > >>>> > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > 36e6795ad3720ba72fc9f4ddabf45f6d67cdc77658a181733fa2c47bfd3799f123b8840a79b538b8504a7b4bdc97fd0b52efab790395a5e022a897bc18405d0c > > > > > > >>>> > > > > > > > > >>>> > > Release notes (in progress) can be found at: > > > > > > >>>> > > > https://accumulo.staged.apache.org/release/accumulo-1.10.3 > > > > > > >>>> > > > > > > > > >>>> > > Release testing instructions: > > > > > > >>>> > > > https://accumulo.apache.org/contributor/verifying-release > > > > > > >>>> > > > > > > > > >>>> > > Please vote one of: > > > > > > >>>> > > [ ] +1 - I have verified and accept... > > > > > > >>>> > > [ ] +0 - I have reservations, but not strong enough to > vote > > > > > > >>>> against... > > > > > > >>>> > > [ ] -1 - Because..., I do not accept... > > > > > > >>>> > > ... these artifacts as the 1.10.3 release of Apache > > > Accumulo. > > > > > > >>>> > > > > > > > > >>>> > > This vote will remain open until at least Mon Apr 10 > > > 08:30:00 > > > > PM > > > > > > UTC > > > > > > >>>> > 2023. > > > > > > >>>> > > (Mon Apr 10 04:30:00 PM EDT 2023 / Mon Apr 10 01:30:00 > PM > > > PDT > > > > > > 2023) > > > > > > >>>> > > Voting can continue after this deadline until the > release > > > > > manager > > > > > > >>>> > > sends an email ending the vote. > > > > > > >>>> > > > > > > > > >>>> > > Thanks! > > > > > > >>>> > > > > > > > > >>>> > > P.S. Hint: download the whole staging repo with > > > > > > >>>> > > wget -erobots=off -r -l inf -np -nH \ > > > > > > >>>> > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > https://repository.apache.org/content/repositories/orgapacheaccumulo-1100/ > > > > > > >>>> > > # note the trailing slash is needed > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>> > > > > > > > > > > > > > > > > > > >
