On 26/02/2019 18:58, Severin Gehwolf wrote:
FWIW, I use 'make test TEST="..."'. Isn't the testing system using that
too?
I am not sure. But I'd expect it to be similar.
[...]
When I was working on that patch years ago, I think one testing machine
got configured in a way so that it wouldn't trivially pass. That may
have changed at some point in time?
Probably.
WRT using the loopback as an alternate configuration,
I wonder - should we add the loopback address before
checking size() < 1?
Not sure how stable the test will be in that case.
It wouldn't help anything. In that case it wouldn't be a regression
test anymore as it would pass *prior* JDK-6425769 *and* after. With a
non-loopback address added too, it would fail prior and pass after.
Fair enough.
Also should we use InetAddress.getLoopbackAddress() instead
of using "127.0.0.1" directly?
InetAddress.getLoopbackAddress() is less deterministic than using IPv4
address. Interestingly the javadoc mentions it may return any loopback
address, not just 127.0.0.1 or ::1. Not sure whether that's a problem
or not, I'd be fine to change that if you prefer.
Not specially - if you believe the test would be less deterministic
that way. The only reason I asked was in the hope to avoid having a hard
dependency on IPv4 vs IPv6 buried in the test code.
So far I think your changes look good.
I am satisfied that they don't trigger intermittent failure,
and I believe these changes are a step in the right direction,
even if the test still passes trivially most of the time.
On a side note: to your knowledge is there anything that would
prevent the test to work meaningfully with the pair:
InetAddress.getLocalHost(), InetAdress.getLoopbackAddress(), after
verifying that InetAddress.getLocalHost().isLoopbackAddress() == false?
I experimented with that - but then the test failed on all
windows machines.
The subprocess seemed to succeed - I could see all the expected
traces saying that the JMX connection worked, but then
ProcessThread.stopProcess() seemed to cause the subprocess to
exit with 1 (instead of the expected 143).
Seems to happen 100% of the times on windows.
On Mac OS X and Linux - I saw it failed too - though with lower
frequency (like 3 or 4 times every 100 runs), and AFAICS
usually later and for a different reason: when it fails, it usually
timeouts when trying to test over SSL.
On Solaris - it seems to fail at least half of the times, usually
in timeout over SSL too.
My guess is that these lower frequency failures might be caused
by reusing the same ports again. We had intermittent test failures
in other networking tests that disappeared once we started binding
explicitly to the loopback address, and letting the system allocate
an ephemeral port (binding with port 0)
That makes me think that the test might have had stability issues if
by misfortune it did not trivially pass.
Maybe this test should really be made manual - @run main/manual [1]?
[1] https://openjdk.java.net/jtreg/tag-spec.html
What are your thoughts?
best regards,
-- daniel
Thanks,
Severin
