Hi Máté, all,
I managed to fight my way though some Apple ID insanity to get a Mac
running with Catalina, Java, Maven and a C compiler. This part of the
puzzle was particularly fun:
https://dd.crosstwine.com/tmp/apple-id.png
Anyway.
> So I prefer to execute the java unit tests using the following command when
> checking RCs:
>
> mvn test -Dsurefire.rerunFailingTestsCount=3 -fae -Pfull-build -DforkCount=1
This command does not work, at least on Catalina—where it forks eight
runners:
% pstree 7846
-+= 07846 administrator /Library/Java/JavaVir.../adoptopenjdk-8...
|-+- 09342 administrator /bin/sh -c cd /Users/...
| \--- 09343 administrator /Library/Java/JavaVir.../adoptopenjdk-8...
|-+- 09640 administrator /bin/sh -c cd /Users/...
| \--- 09641 administrator /Library/Java/JavaVir.../adoptopenjdk-8...
|-+- 09797 administrator /bin/sh -c cd /Users/...
| \--- 09798 administrator /Library/Java/JavaVir.../adoptopenjdk-8...
|-+- 09893 administrator /bin/sh -c cd /Users/...
| \--- 09898 administrator /Library/Java/JavaVir.../adoptopenjdk-8...
|-+- 09912 administrator /bin/sh -c cd /Users/...
| \--- 09913 administrator /Library/Java/JavaVir.../adoptopenjdk-8...
|-+- 09942 administrator /bin/sh -c cd /Users/...
| \--- 09944 administrator /Library/Java/JavaVir.../adoptopenjdk-8...
|-+- 09955 administrator /bin/sh -c cd /Users/...
| \--- 09956 administrator /Library/Java/JavaVir.../adoptopenjdk-8...
\-+- 09960 administrator /bin/sh -c cd /Users/...
\--- 09961 administrator /Library/Java/JavaVir.../adoptopenjdk-8...
It seems forkCount is overriden by the surefire-forkcount property.
This seems to do what you expect:
mvn test -Dsurefire.rerunFailingTestsCount=3 -fae -Pfull-build
-Dsurefire-forkcount=1
But independently of that, I can reproduce the issue—even with a "real"
sequential run. It does not always result in the stack trace you saw,
but often hangs with "funny" errors such as:
[WARNING] Corrupted STDOUT by directly writing to native stream in forked
JVM 1. See FAQ web page and the dump file
/Users/administrator/zookeeper/zookeeper-server/target/surefire-reports/2021-01-28T17-56-26_094-jvmRun1.dumpstream
# Created at 2021-01-28T18:37:51.381
Corrupted STDOUT by directly writing to native stream in forked JVM 1.
Stream '[59.370s][warning][os,thread] Failed to start thread - pthread_create
failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 4k, detached.'.
I'm planning to have a closer look soon.
Cu, -D
P.-S. — Enrico: I am also able to reproduce the C compilation issues you
reported.
--8<---------------original message------------->8---
Szalay-Bekő Máté <szalay.beko.m...@gmail.com> writes:
> this is not specific to 3.7, but in general:
> from time to time I see different tests failing intermittently. I assume
> this can be caused by other tests running in parallel. (because re-running
> those tests alone usually makes them succeed)
> So I prefer to execute the java unit tests using the following command when
> checking RCs:
>
> mvn test -Dsurefire.rerunFailingTestsCount=3 -fae -Pfull-build -DforkCount=1
>
> As far as I remember, by default the fork count is set to 8 in maven, and
> the CI is using (or was using before the migration) -DforkCount=4.
>
>
> Regards,
> Mate
>
> On Mon, Jan 25, 2021 at 1:51 PM Flavio Junqueira <f...@apache.org> wrote:
>
>> I don't want to mess up with the vote thread, so I'm responding to this
>> separately. I have been trying to build locally too unsuccessfully. I've
>> been trying on an Ubuntu VM, Java 8 (build 1.8.0_181-b13), Maven 3.6.0. The
>> set of tests failing varies from build to build, if it makes sense, I can
>> try to collect all test failures I have seen and post.
>>
>> -Flavio
>>
>> > On 25 Jan 2021, at 13:38, Szalay-Bekő Máté <szalay.beko.m...@gmail.com>
>> wrote:
>> >
>> > +0 (and not even binding :) )
>> >
>> > - I built the source code (-Pfull-build) on Ubuntu 18.04.3 using OpenJDK
>> > 8u265 and maven 3.6.3.
>> > - I also built and executed unit tests for zkpython
>> > - the unit tests passed for the C-client and for python client
>> > - checkstyle and spotbugs passed
>> > - apache-rat passed
>> > - owasp (CVE check) passed
>> > - fatjar built (-Pfatjar)
>> > - I executed a quick rolling-upgrade test from 3.5.9 and from 3.6.2.
>> (using
>> > https://github.com/symat/zk-rolling-upgrade-test)
>> >
>> > for some reason the java unit tests failed for me.
>> >
>> > On mac (jdk 1.8.212 and maven 3.6.3), I got all the unit tests executed
>> > successfully, but then the maven job still failed for hbase-server test
>> > with error message (with -DforkCount=4 and even with -DforkCount=1) like:
>> > -------------
>> > [ERROR] ExecutionException There was an error in the forked process
>> > [ERROR] unable to create new native thread
>> > [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException:
>> > ExecutionException There was an error in the forked process
>> > [ERROR] unable to create new native thread
>> > [ERROR] at
>> >
>> org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:510)
>> > -------------
>> >
>> >
>> > Then I tried on a dockerized environment (ubuntu 18.4, OpenJDK 8u265 and
>> > maven 3.6.3) and I got other kinds of strange maven errors:
>> > -------------------
>> > [ERROR] Caused by:
>> > org.apache.maven.surefire.booter.SurefireBooterForkException: The forked
>> VM
>> > terminated without properly saying goodbye. VM crash or System.exit
>> called?
>> > [ERROR] Command was /bin/sh -c cd
>> > /tmp/zk/apache-zookeeper-3.7.0/zookeeper-server &&
>> > /home/symat/.sdkman/candidates/java/8.0.265-open/jre/bin/java -Xmx512m
>> > -Dtest.junit.threads=8 -Dzookeeper.junit.threadid=3
>> >
>> -javaagent:/home/symat/.m2/repository/org/jmockit/jmockit/1.48/jmockit-1.48.jar
>> > -jar
>> >
>> /tmp/zk/apache-zookeeper-3.7.0/zookeeper-server/target/surefire/surefirebooter8828313385463488429.jar
>> > /tmp/zk/apache-zookeeper-3.7.0/zookeeper-server/target/surefire
>> > 2021-01-25T11-54-03_621-jvmRun3 surefire4024538135165099286tmp
>> > surefire_37800399112966511000tmp
>> > [ERROR] Process Exit Code: 0
>> > [ERROR] at
>> >
>> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:669)
>> > [ERROR] at
>> >
>> org.apache.maven.plugin.surefire.booterclient.ForkStarter.access$600(ForkStarter.java:115)
>> > [ERROR] at
>> >
>> org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:444)
>> > [ERROR] at
>> >
>> org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:420)
>> > [ERROR] at
>> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> > [ERROR] at
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> > [ERROR] at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> > [ERROR] at java.lang.Thread.run(Thread.java:748)
>> > [ERROR]
>> > --------------------------------------
>> >
>> >
>> > These issues might be very well specific to my local (mac or docker on
>> mac)
>> > environments. This is why I didn't vote with -1
>> > Can someone else run the java unit tests successfully locally?
>> >
>> > I also tried to check if the CI was green for the last PR on 3.7.0 (
>> > https://github.com/apache/zookeeper/pull/1586/checks), but it looks the
>> CI
>> > haven't even started to execute the tests, due to errors in the "install
>> C
>> > dependencies" step.
>> >
>> > Regards,
>> > Mate
>> >
>> > On Sun, Jan 24, 2021 at 11:39 PM Patrick Hunt <ph...@apache.org> wrote:
>> >
>> >> +1. xsum/sig verified. rat ran clean. built and dependency checks are
>> fine.
>> >> Tried running some manual clusters and it was successful.
>> >>
>> >> Regards,
>> >>
>> >> Patrick
>> >>
>> >>
>> >> On Sun, Jan 24, 2021 at 12:11 PM Damien Diederen <ddiede...@apache.org>
>> >> wrote:
>> >>
>> >>>
>> >>> Dear all,
>> >>>
>> >>> This is a second release candidate for ZooKeeper 3.7.0. Compared to
>> >>> RC0, it fixes a tarball generation issue, includes a description of the
>> >>> 'whoami' CLI command, and incorporates a contribution to ZooInspector.
>> >>>
>> >>> ZooKeeper 3.7.0 introduces a number of new features, notably:
>> >>>
>> >>> * An API to start a ZooKeeper server from Java (ZOOKEEPER-3874);
>> >>>
>> >>> * Quota enforcement (ZOOKEEPER-3301);
>> >>>
>> >>> * Host name canonicalization in quorum SASL authentication
>> >>> (ZOOKEEPER-4030);
>> >>>
>> >>> * Support for BCFKS key/trust store format (ZOOKEEPER-3950);
>> >>>
>> >>> * A choice of mandatory authentication scheme(s) (ZOOKEEPER-3561);
>> >>>
>> >>> * A "whoami" API and CLI command (ZOOKEEPER-3969);
>> >>>
>> >>> * The possibility of disabling digest authentication (ZOOKEEPER-3979);
>> >>>
>> >>> * Multiple SASL "superUsers" (ZOOKEEPER-3959);
>> >>>
>> >>> * Fast-tracking of throttled requests (ZOOKEEPER-3683);
>> >>>
>> >>> * Additional security metrics (ZOOKEEPER-3978);
>> >>>
>> >>> * SASL support in the C and Perl clients (ZOOKEEPER-1112,
>> >>> ZOOKEEPER-3714);
>> >>>
>> >>> * A new zkSnapshotComparer.sh tool (ZOOKEEPER-3427);
>> >>>
>> >>> * Notes on how to benchmark ZooKeeper with the YCSB tool
>> >>> (ZOOKEEPER-3264).
>> >>>
>> >>> The release notes are available here:
>> >>>
>> >>>
>> >>>
>> >>
>> https://people.apache.org/~ddiederen/zookeeper-3.7.0-candidate-1/website/releasenotes.html
>> >>>
>> >>>
>> >>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310801&version=12346617
>> >>>
>> >>> *** Please download, test and vote by January 31st 2020, 23:59 UTC+0.
>> ***
>> >>>
>> >>> Source files:
>> >>>
>> >>> https://people.apache.org/~ddiederen/zookeeper-3.7.0-candidate-1/
>> >>>
>> >>> Maven staging repo:
>> >>>
>> >>>
>> >>>
>> >>
>> https://repository.apache.org/content/repositories/orgapachezookeeper-1066/
>> >>>
>> >>> The release candidate tag in git to be voted upon: release-3.7.0-1
>> >>>
>> >>> https://github.com/apache/zookeeper/tree/release-3.7.0-1
>> >>>
>> >>> ZooKeeper's KEYS file containing PGP keys we use to sign the release:
>> >>>
>> >>> https://www.apache.org/dist/zookeeper/KEYS
>> >>>
>> >>> The staging version of the website is:
>> >>>
>> >>>
>> >>>
>> >>
>> https://people.apache.org/~ddiederen/zookeeper-3.7.0-candidate-1/website/
>> >>>
>> >>> Should we release this candidate?
>> >>>
>> >>> Damien Diederen
>> >>>
>> >>
>>
>>
