Thanks. I'll follow up on that issue.
On Mon, Aug 31, 2020 at 6:18 PM Joey Frazee <[email protected]> wrote: > > Thanks for the reply. I opened #1696. > > I first saw this on the candidate source build, but AFAICT it applies to the > convenience binary posted too, Java 8 and 11 runtimes, and different ways of > deploying. > > -joey > On Aug 31, 2020, 12:05 PM -0700, Christopher <[email protected]>, wrote: > > Joey, was the IllegalAccessException a warning, or did it cause a failure? > > Also, what version of Java were you using? Some newer versions of Java > > report some IllegalAccess issues when some older code tries to access > > unsafe internal areas of the JVM, and there's little we can do about > > that, except to slowly migrate to newer libraries. > > > > Please create a new issue with the details, if you can reproduce the > > issue you saw. > > > > In general, some amount of dependency convergence is expected > > downstream, especially before using the software in production, > > because we can't anticipate or test all class path possibilities. This > > is one reason (of many) that the ASF releases (buildable, packageable, > > and redistributable) source code, and doesn't make guarantees > > regarding binaries (only sometimes building reference binaries for > > convenience, a practice that varies between Apache projects). > > Out-of-the box, the Accumulo PMC tries to make Accumulo work with at > > least *some* dependency versions, but may not catch all dependency > > issues. It is expected that downstream users will take some > > responsibility for configuring their class path appropriately for > > their own environment. But, if you report a specific incompatibility > > or bug, we may be able to accept a patch or pull request to fix it, or > > we may be able to fix it ourselves upon receiving the information. > > > > > > On Mon, Aug 31, 2020 at 2:35 PM Joey Frazee > > <[email protected]> wrote: > > > > > > I wouldn’t necessarily consider it a problem, but I thought I’d mention > > > that while testing I ran into an IlegalAccessException in the shell if > > > ./hadoop/common/lib/* is added to the classpath because the commons-* > > > versions are ahead of the Hadoop releases now (incl. both 2.10.x and > > > 3.2.x) [1, 2]. > > > > > > Arguably, someone shouldn’t include all of ./hadoop/common/lib/* and it’s > > > easily fixable by excluding commons-cli, but it’s a thing. > > > > > > 1. https://github.com/apache/accumulo/pull/1687 > > > 2. https://github.com/apache/accumulo/pull/1688 > > > > > > -joey > > > On Aug 31, 2020, 10:21 AM -0700, Christopher <[email protected]>, wrote: > > > > I wouldn't personally consider #1695 a blocker. The fix for that can > > > > make it into 1.10.1 if the re-testing of continuous ingest goes well. > > > > > > > > On Mon, Aug 31, 2020 at 12:46 PM Jeffrey Manno > > > > <[email protected]> wrote: > > > > > > > > > > I don't recall remembering to turn off ingest before running > > > > > verification. > > > > > This is my first time running the tests with these scripts as before > > > > > I was > > > > > using accumulo-testing to handle most of this. So that is definitely > > > > > my > > > > > fault on failing to make sure to do that because I was aware I needed > > > > > too. > > > > > On a further note, I realize I also didn't run the agitator properly. > > > > > I > > > > > confirmed now, using the same method as Karthick used, on how to run > > > > > it as > > > > > expected. Again, my apologies there. I am re-running things now as > > > > > well to > > > > > get more accurate results. > > > > > > > > > > I did run into the zookeeper issue Keither mentioned and the bad > > > > > options > > > > > error he mentioned in #1695. > > > > > > > > > > On Mon, Aug 31, 2020 at 12:37 PM Christopher <[email protected]> > > > > > wrote: > > > > > > > > > > > Okay, thanks for the update. I was just about to call the vote on > > > > > > Mike's behalf (because the vote is currently tied, and can't pass > > > > > > as-is and this issue needed to be checked first). Mike is > > > > > > unavailable > > > > > > today due to personal matters, so would be unlikely to call the vote > > > > > > himself. However, based on this new information, I will wait until > > > > > > you > > > > > > can report back. > > > > > > > > > > > > Jeffrey Manno: you said you were able to observe the same issue that > > > > > > Keith and Karthick saw. Can you check again to see if your testing > > > > > > had > > > > > > the same issue where verification ran before ingest was stopped? > > > > > > > > > > > > On Mon, Aug 31, 2020 at 12:30 PM Keith Turner <[email protected]> > > > > > > wrote: > > > > > > > > > > > > > > I started investigating this today. I quickly noticed two things. > > > > > > > First when I scanned the table for some of the data reported as > > > > > > > missing by the MR job it was there. Second the range of the > > > > > > > supposed > > > > > > > missing data covered the entire table. Then I realized we never > > > > > > > stopped ingest, so during verification ingest was still running. > > > > > > > This > > > > > > > would cause data to report as missing because mappers read > > > > > > > different > > > > > > > parts of the table at different times. > > > > > > > > > > > > > > I was advising Karthick on how to run the test and I gave some > > > > > > > really > > > > > > > bad advice, I forgot to mention to stop ingest. Sorry everyone, > > > > > > > please forgive my blunder. > > > > > > > > > > > > > > I stopped ingest and started the verification job again. It has > > > > > > > more > > > > > > > data to verify than last time, there are 71B entries in the table > > > > > > > now. > > > > > > > I also enabled snappy compression of map output as a precaution to > > > > > > > ensure there is enough space. Was planning on less data when > > > > > > > allocating the cluster, but I think there is enough space to run > > > > > > > the > > > > > > > job though. The cluster has 9TB of 25TB used. I'll report back > > > > > > > when > > > > > > > the job finishes. > > > > > > > > > > > > > > > > > > > > > On Sun, Aug 30, 2020 at 12:07 PM karthick rn > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > > > -1 > > > > > > > > > > > > > > > > Keith and I tested the continuous ingest with agitation on 11 > > > > > > > > nodes (9 > > > > > > > > workers) for 24 hrs and noticed the following after running the > > > > > > MapReduce > > > > > > > > verify job. The "undefined" counter is greater than 0 may be > > > > > > > > indicating > > > > > > > > data loss. > > > > > > > > > > > > > > > > org.apache.accumulo.test.continuous.ContinuousVerify$Counts > > > > > > > > REFERENCED=35470816664 > > > > > > > > * UNDEFINED=707335949* > > > > > > > > UNREFERENCED=715995424 > > > > > > > > > > > > > > > > The cluster was setup using Muchos ( > > > > > > https://github.com/apache/fluo-muchos) > > > > > > > > & following are the details > > > > > > > > > > > > > > > > * 11 Azure VMs (Standard D8s) > > > > > > > > * Managed Disk – 3 x 1024g per VM > > > > > > > > * Hadoop 2.10.0 > > > > > > > > * Accumulo 1.10.0-rc2 > > > > > > > > * Java 8 > > > > > > > > * CentOS7.5 > > > > > > > > > > > > > > > > Created the table 'ci' and split using the below commands > > > > > > > > $ accumulo org.apache.accumulo.test.continuous.GenSplits 90 > > > > > > > splits.txt > > > > > > > > $ accumulo shell -u root -p secret -e 'createtable ci -sf > > > > > > > > splits.txt' > > > > > > > > > > > > > > > > Below is the agitator options used for the test > > > > > > > > nohup ./tserver-agitator.pl 1:10 1:10 1 3 > logs/tserver-ag.out > > > > > > > > 2> > > > > > > > > logs/tserver-ag.err & > > > > > > > > > > > > > > > > We will investigate the data loss and share our findings. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Karthick > > > > > > > > > > > > > > > > On Sat, 29 Aug 2020 at 23:32, Keith Turner <[email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Karthick and I are working together to run random walk and > > > > > > > > > continuous > > > > > > > > > ingest on two clusters using this RC. After continuous ingest > > > > > > > > > ran > > > > > > for > > > > > > > > > 24 hrs we tried to start verification and ran into the > > > > > > > > > following > > > > > > > > > issue. I plan to vote after the test completes, but wanted to > > > > > > > > > let > > > > > > > > > anyone else running continuous ingest know about this. > > > > > > > > > > > > > > > > > > https://github.com/apache/accumulo/issues/1695 > > > > > > > > > > > > > > > > > > We also ran into another issue w/ the verification script > > > > > > > > > related to > > > > > > > > > ZK 3.5 that we worked around w/ a hack (Karthick copied the > > > > > > > > > ZK jar > > > > > > > > > from $ZK_HOME/lib to $ZK_HOME so the script could find the > > > > > > > > > jars :). > > > > > > > > > Need to open an issue about this too. > > > > > > > > > > > > > > > > > > On Thu, Aug 27, 2020 at 12:36 PM Mike Miller > > > > > > > > > <[email protected]> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Accumulo Developers, > > > > > > > > > > > > > > > > > > > > Please consider the following candidate for Apache Accumulo > > > > > > > > > > 1.10.0. > > > > > > > > > > > > > > > > > > > > Git Commit: > > > > > > > > > > 4d261254c3ac43a3bd13ce974e91ce4303a83998 > > > > > > > > > > Branch: > > > > > > > > > > 1.10.0-rc2 > > > > > > > > > > > > > > > > > > > > If this vote passes, a gpg-signed tag will be created using: > > > > > > > > > > git tag -f -m 'Apache Accumulo 1.10.0' -s rel/1.10.0 \ > > > > > > > > > > 4d261254c3ac43a3bd13ce974e91ce4303a83998 > > > > > > > > > > > > > > > > > > > > Staging repo: > > > > > > > > > > > > > > > > > > > > > > > > > https://repository.apache.org/content/repositories/orgapacheaccumulo-1086 > > > > > > > > > > Source (official release artifact): > > > > > > > > > > > > > > > > > > > > > > > > > https://repository.apache.org/content/repositories/orgapacheaccumulo-1086/org/apache/accumulo/accumulo/1.10.0/accumulo-1.10.0-src.tar.gz > > > > > > > > > > Binary: > > > > > > > > > > > > > > > > > > > > > > > > > https://repository.apache.org/content/repositories/orgapacheaccumulo-1086/org/apache/accumulo/accumulo/1.10.0/accumulo-1.10.0-bin.tar.gz > > > > > > > > > > > > > > > > > > > > Append ".asc" to download the cryptographic signature for a > > > > > > > > > > given > > > > > > > > > artifact. > > > > > > > > > > (You can also append ".sha1" or ".md5" instead in order to > > > > > > > > > > verify > > > > > > the > > > > > > > > > > checksums > > > > > > > > > > generated by Maven to verify the integrity of the Nexus > > > > > > > > > > repository > > > > > > > > > staging > > > > > > > > > > area.) > > > > > > > > > > > > > > > > > > > > Signing keys are available at > > > > > > https://www.apache.org/dist/accumulo/KEYS > > > > > > > > > > (Expected fingerprint: > > > > > > > > > > 1914AF6FE2C53672C87CE1DADC8FFDC342894E89) > > > > > > > > > > > > > > > > > > > > In addition to the tarballs and their signatures, the > > > > > > > > > > following > > > > > > checksum > > > > > > > > > > files will be added to the dist/release SVN area after > > > > > > > > > > release: > > > > > > > > > > accumulo-1.10.0-src.tar.gz.sha512 will contain: > > > > > > > > > > SHA512 (accumulo-1.10.0-src.tar.gz) = > > > > > > > > > > > > > > > > > > > > > > > > > 81f2a8f8273e2bdfe46d6a807dc38276ee2937ced648829648b7750bfc22816c13d43461d1b08c50a6957d78a999ae3109c93d2f31c7d8be116e91e0ea25f5c2 > > > > > > > > > > accumulo-1.10.0-bin.tar.gz.sha512 will contain: > > > > > > > > > > SHA512 (accumulo-1.10.0-bin.tar.gz) = > > > > > > > > > > > > > > > > > > > > > > > > > 9d3023c8724069282035ed6dcb047f737c1c53dc05f7b15da2cfd941f51d1d7720892496475430eb639f3a36c83f4eecc1942c0317c67d38dcf2061d06beb648 > > > > > > > > > > > > > > > > > > > > Release notes (in progress) can be found at: > > > > > > > > > > https://accumulo.apache.org/release/accumulo-1.10.0/ > > > > > > > > > > > > > > > > > > > > Release testing instructions: > > > > > > > > > > https://accumulo.apache.org/contributor/verifying-release > > > > > > > > > > > > > > > > > > > > Please vote one of: > > > > > > > > > > [ ] +1 - I have verified and accept... > > > > > > > > > > [ ] +0 - I have reservations, but not strong enough to vote > > > > > > against... > > > > > > > > > > [ ] -1 - Because..., I do not accept... > > > > > > > > > > ... these artifacts as the 1.10.0 release of Apache > > > > > > > > > > Accumulo. > > > > > > > > > > > > > > > > > > > > This vote will remain open until at least Sun Aug 30 > > > > > > > > > > 16:30:00 UTC > > > > > > 2020. > > > > > > > > > > (Sun Aug 30 12:30:00 EDT 2020 / Sun Aug 30 09:30:00 PDT > > > > > > > > > > 2020) > > > > > > > > > > Voting can continue after this deadline until the release > > > > > > > > > > manager > > > > > > > > > > sends an email ending the vote. > > > > > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > > > > > P.S. Hint: download the whole staging repo with > > > > > > > > > > wget -erobots=off -r -l inf -np -nH \ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://repository.apache.org/content/repositories/orgapacheaccumulo-1086/ > > > > > > > > > > # note the trailing slash is needed > > > > > > > > > > > > > > >
