If I update the pom's with the commons-configuration changes suggested by Dave in a previous post, I can successfully run the Sunny profile tests with both Hadoop 2 and Hadoop 3.
On Tue, Apr 11, 2023 at 1:16 PM Daniel Roberts <ddani...@gmail.com> wrote: > Yes, I changed the memory settings last week and was able to pass the > ExamplesIT.testScansWithInterference and > ExamplesIT.testIsolatedScansWithInterference > without issue. Given these environmental tweaks, do we have a documented > minimum machine spec that we expect the release tests to pass on? > > On Tue, Apr 11, 2023 at 1:00 PM Christopher Shannon < > christopher.l.shan...@gmail.com> wrote: > > > The test failure for ExamplesIT.testScansWithInterference is likely from > > out of memory. See https://github.com/apache/accumulo/issues/3281 > > > > On Tue, Apr 11, 2023 at 12:21 PM Dominic Garguilo < > > dominic.gargu...@gmail.com> wrote: > > > > > After running > > > mvn clean verify -Phadoop3,sunny > > > I am consistently seeing the following: > > > > > > [ERROR] > > > > > > > > > > org.apache.accumulo.core.conf.CredentialProviderFactoryShimTest.extractFromHdfs > > > > Time elapsed: 3.605 s <<< ERROR! > > > > java.lang.NoClassDefFoundError: > > > > org/apache/commons/beanutils/BeanIntrospector > > > > at > > > > > > > > > > org.apache.accumulo.core.conf.CredentialProviderFactoryShimTest.extractFromHdfs(CredentialProviderFactoryShimTest.java:185) > > > > Caused by: java.lang.ClassNotFoundException: > > > > org.apache.commons.beanutils.BeanIntrospector > > > > at > > > > > > > > > > org.apache.accumulo.core.conf.CredentialProviderFactoryShimTest.extractFromHdfs(CredentialProviderFactoryShimTest.java:185) > > > > > > > > > This test passes consistently when run in my IDE but fails every time > > when > > > running the mentioned maven command. > > > > > > On Mon, Apr 10, 2023 at 6:04 PM Dave Marion <dmario...@gmail.com> > wrote: > > > > > > > *mvn clean verify -Phadoop3,sunny* failed in ExamplesIT with the > > > > aforementioned changes. Specifically, > > > ExamplesIT.testScansWithInterference > > > > failed with a server error when closing the batch writer. No further > > > > information in the logs. > > > > > > > > On Mon, Apr 10, 2023 at 5:06 PM Dave Marion <dmario...@gmail.com> > > wrote: > > > > > > > > > Updating commons-configuration from 1.6 to 1.10 did the trick I > > think. > > > > > Versions 1.7, 1.8, and 1.9 had compilation issues. I'm not veto-ing > > > > because > > > > > other people have built with the hadoop3 profile even though more > > than > > > > one > > > > > of us have had an issue. Here are the changes I made to get me past > > the > > > > > CredentialProviderFactoryShimTest test failure in conf using the > > > hadoop3 > > > > > profile. > > > > > > > > > > > > > > > diff --git a/core/pom.xml b/core/pom.xml > > > > > index 163a1c062f..2aad10486a 100644 > > > > > --- a/core/pom.xml > > > > > +++ b/core/pom.xml > > > > > @@ -201,6 +201,8 @@ > > > > > > > > > > > > > > > > > > > > <allow>org[.]apache[.]hadoop[.]mapred[.](FileAlreadyExistsException|InvalidJobConfException)</allow> > > > > > <!-- allow lexicoders to throw iterator exceptions > > --> > > > > > > > > > > > > > > > > > > > > <allow>org[.]apache[.]accumulo[.]core[.]iterators[.]ValueFormatException</allow> > > > > > + <!-- allow commons-configuration for deprecated > > > > > ClientConfiguration constructor --> > > > > > + > > > > > > > <allow>org[.]apache[.]commons[.]configuration[.]Configuration</allow> > > > > > </allows> > > > > > </configuration> > > > > > </execution> > > > > > diff --git a/pom.xml b/pom.xml > > > > > index 854443da2d..2193b46b54 100644 > > > > > --- a/pom.xml > > > > > +++ b/pom.xml > > > > > @@ -212,7 +212,7 @@ > > > > > <dependency> > > > > > <groupId>commons-configuration</groupId> > > > > > <artifactId>commons-configuration</artifactId> > > > > > - <version>1.6</version> > > > > > + <version>1.10</version> > > > > > </dependency> > > > > > <dependency> > > > > > <groupId>commons-io</groupId> > > > > > > > > > > > > > > > > > > > > On Mon, Apr 10, 2023 at 12:16 PM Dave Marion <dmario...@gmail.com> > > > > wrote: > > > > > > > > > >> From what I can tell: > > > > >> > > > > >> 1. commons-configuration:1.6 has a commons-beanutils:1.7.0 > > dependency > > > > >> 2. hadoop-common:3.0.3 has a commons-beanutils:1.9.3 dependency, > > > > >> hadoop-common:2.6.5 has no such dependency > > > > >> > > > > >> Hadoop 3.0.3 is failing because the commons-beanutils:1.9.3 > > dependency > > > > >> must not be on the classpath for the test. I have no idea why this > > is > > > > >> failing on my machine and not for others. I'm also seeing general > > > > slowness > > > > >> building 1.10.3 on my machine, but not building other git > branches. > > > > >> > > > > >> On Mon, Apr 10, 2023 at 8:24 AM Dave Marion <dmario...@gmail.com> > > > > wrote: > > > > >> > > > > >>> I built the source tarball using the hadoop3 profile again and > > > > >>> encountered the issue below. The version of hadoop (3.0.3) has > not > > > > changed, > > > > >>> so I assume that this is related to a change in our dependencies. > > > > >>> > > > > >>> [ERROR] Tests run: 10, Failures: 0, Errors: 1, Skipped: 0, Time > > > > elapsed: > > > > >>> 3.856 s <<< FAILURE! - in > > > > >>> org.apache.accumulo.core.conf.CredentialProviderFactoryShimTest > > > > >>> [ERROR] > > > > >>> > > > > > > > > > > org.apache.accumulo.core.conf.CredentialProviderFactoryShimTest.extractFromHdfs > > > > >>> Time elapsed: 1.186 s <<< ERROR! > > > > >>> java.lang.NoClassDefFoundError: > > > > >>> org/apache/commons/beanutils/BeanIntrospector > > > > >>> at > > > > >>> > > > > > > > > > > org.apache.accumulo.core.conf.CredentialProviderFactoryShimTest.extractFromHdfs(CredentialProviderFactoryShimTest.java:185) > > > > >>> Caused by: java.lang.ClassNotFoundException: > > > > >>> org.apache.commons.beanutils.BeanIntrospector > > > > >>> at > > > > >>> > > > > > > > > > > org.apache.accumulo.core.conf.CredentialProviderFactoryShimTest.extractFromHdfs(CredentialProviderFactoryShimTest.java:185) > > > > >>> > > > > >>> On Sun, Apr 9, 2023 at 11:41 AM Jeffrey Manno < > > > > jeffreymann...@gmail.com> > > > > >>> wrote: > > > > >>> > > > > >>>> +1 > > > > >>>> > > > > >>>> I did run into a few flaky tests but none seem to warrant a > > stoppage > > > > to > > > > >>>> this release candidate. Two common ones are SuspendTabletsIT and > > > > >>>> DeleteTableDuringSplitIT. > > > > >>>> > > > > >>>> Other checks I did: > > > > >>>> > > > > >>>> - Ran the entire test suite, along with sunny day tests. > > > > >>>> - Verified checksums. > > > > >>>> - Did some small ingest testing with agitation. > > > > >>>> > > > > >>>> > > > > >>>> On Sat, Apr 8, 2023 at 9:18 AM Christopher Shannon < > > > > >>>> christopher.l.shan...@gmail.com> wrote: > > > > >>>> > > > > >>>> > +1 (binding), LGTM > > > > >>>> > > > > > >>>> > Some of the things I did for verification/testing locally: > > > > >>>> > > > > > >>>> > * Validated signatures and checksums > > > > >>>> > * Verified license and notice files in archives > > > > >>>> > * Verified source license headers with 'mvn apache-rat:check' > > > > >>>> > * Built and ran all the sunny integration tests > > > > >>>> > * Started up using Uno to make sure everything started > correctly > > > > >>>> (also ran > > > > >>>> > a couple scans, etc) > > > > >>>> > > > > > >>>> > On Fri, Apr 7, 2023 at 4:23 PM Christopher < > ctubb...@apache.org > > > > > > > >>>> wrote: > > > > >>>> > > > > > >>>> > > Accumulo Developers, > > > > >>>> > > > > > > >>>> > > Please consider the following candidate for Apache Accumulo > > > > 1.10.3. > > > > >>>> > > > > > > >>>> > > Git Commit: > > > > >>>> > > 733863638d85d0109d217da7ea5f36f5e483e207 > > > > >>>> > > Branch: > > > > >>>> > > 1.10.3-rc1 > > > > >>>> > > > > > > >>>> > > If this vote passes, a gpg-signed tag will be created using: > > > > >>>> > > git tag -f -s -m 'Apache Accumulo 1.10.3' rel/1.10.3 \ > > > > >>>> > > 733863638d85d0109d217da7ea5f36f5e483e207 > > > > >>>> > > > > > > >>>> > > Staging repo: > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > > > > > > > https://repository.apache.org/content/repositories/orgapacheaccumulo-1100 > > > > >>>> > > Source (official release artifact): > > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > > > > > > > https://repository.apache.org/content/repositories/orgapacheaccumulo-1100/org/apache/accumulo/accumulo/1.10.3/accumulo-1.10.3-src.tar.gz > > > > >>>> > > Binary: > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > > > > > > > https://repository.apache.org/content/repositories/orgapacheaccumulo-1100/org/apache/accumulo/accumulo/1.10.3/accumulo-1.10.3-bin.tar.gz > > > > >>>> > > > > > > >>>> > > Append ".asc" to download the cryptographic signature for a > > > given > > > > >>>> > artifact. > > > > >>>> > > (You can also append ".sha1" or ".md5" instead in order to > > > verify > > > > >>>> the > > > > >>>> > > checksums > > > > >>>> > > generated by Maven to verify the integrity of the Nexus > > > repository > > > > >>>> > > staging area.) > > > > >>>> > > > > > > >>>> > > Signing keys are available at > > > > >>>> https://www.apache.org/dist/accumulo/KEYS > > > > >>>> > > (Expected fingerprint: > > 8CC4F8A2B29C2B040F2B835D6F0CDAE700B6899D) > > > > >>>> > > > > > > >>>> > > In addition to the tarballs and their signatures, the > > following > > > > >>>> checksum > > > > >>>> > > files will be added to the dist/release SVN area after > > release: > > > > >>>> > > accumulo-1.10.3-src.tar.gz.sha512 will contain: > > > > >>>> > > SHA512 (accumulo-1.10.3-src.tar.gz) = > > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > > > > > > > 436707da7424ea1b7993355d70747068264276234a4772f9da3866a638ad7860a754436f34ab2108e54801cced67f00ca2cfe0f46dfb80f2d7609b7b7112045e > > > > >>>> > > accumulo-1.10.3-bin.tar.gz.sha512 will contain: > > > > >>>> > > SHA512 (accumulo-1.10.3-bin.tar.gz) = > > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > > > > > > > 36e6795ad3720ba72fc9f4ddabf45f6d67cdc77658a181733fa2c47bfd3799f123b8840a79b538b8504a7b4bdc97fd0b52efab790395a5e022a897bc18405d0c > > > > >>>> > > > > > > >>>> > > Release notes (in progress) can be found at: > > > > >>>> > > https://accumulo.staged.apache.org/release/accumulo-1.10.3 > > > > >>>> > > > > > > >>>> > > Release testing instructions: > > > > >>>> > > https://accumulo.apache.org/contributor/verifying-release > > > > >>>> > > > > > > >>>> > > Please vote one of: > > > > >>>> > > [ ] +1 - I have verified and accept... > > > > >>>> > > [ ] +0 - I have reservations, but not strong enough to vote > > > > >>>> against... > > > > >>>> > > [ ] -1 - Because..., I do not accept... > > > > >>>> > > ... these artifacts as the 1.10.3 release of Apache > Accumulo. > > > > >>>> > > > > > > >>>> > > This vote will remain open until at least Mon Apr 10 > 08:30:00 > > PM > > > > UTC > > > > >>>> > 2023. > > > > >>>> > > (Mon Apr 10 04:30:00 PM EDT 2023 / Mon Apr 10 01:30:00 PM > PDT > > > > 2023) > > > > >>>> > > Voting can continue after this deadline until the release > > > manager > > > > >>>> > > sends an email ending the vote. > > > > >>>> > > > > > > >>>> > > Thanks! > > > > >>>> > > > > > > >>>> > > P.S. Hint: download the whole staging repo with > > > > >>>> > > wget -erobots=off -r -l inf -np -nH \ > > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > > > > > > > https://repository.apache.org/content/repositories/orgapacheaccumulo-1100/ > > > > >>>> > > # note the trailing slash is needed > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>> > > > > > > > > > >
