[
https://issues.apache.org/jira/browse/SOLR-13747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-13747:
----------------------------
Attachment: SOLR-13747.patch
Status: Open (was: Open)
Some background...
In SOLR-12988, during the dicsussion of re-enabling SSL testing under java11
knowing that some java 11 versions were broken, I made the following comments...
{quote}
(on the Junit tests side, having assumes around JVM version is fine – because
even then it's not a "silent" behavior change, it's an explicitly "test ignored
because XYZ")
{quote}
{quote}
if devs are running tests with a broken JVM, then the tests can & should fail
... that's the job of the tests. it's a bad idea to make the tests "hide" the
failure by "faking" that things work using a degraded cipher, or skipping SSL
completely (yes, i also think mark's changes to SSLTestConfig in December as
part of his commit on this issue was a terrible idea as well) ... the *ONLY*
thing we should _consider_ allowing tests to change about their behavior if
they see a JVM is "broken" is to SKIP ie:
assume(SomethingThatIsFalseForTheBrokenJVM)
{quote}
Ultimately, adding an {{SSLTestConfig.assumeSslIsSafeToTest()}} method seemed
better then doing a hard {{fail(..)}} in any test that wanted to use SSL --
particularly once we realized that (at that time) every available version of
Java 13 was affected by SSL bugs. {{SKIP}} ing tests (instead of failing
outright) ment we could still have jenkins jobs running the latest jdk13-ea
available looking for _other_ bugs, w/o getting noise due to known SSL bugs.
But the fact that SOLR-13746 slipped through the cracks has caused me to
seriously regret that decision -- and lead me to wonder:
* Do we have committers who are _still_ running {{ant test}} with "bad" JDKs
that don't realize thousands of tests are getting skipped?
* What if down the road a jenkins node gets rebuilt/reverted to use an older
jdk11 version, would anyone notice?
----
The attached patch adds a new
{{TestSSLTestConfig.testFailIfUserRunsTestsWithJVMThatHasKnownSSLBugs}} to the
{{solr/test-framework}} module that does what i's name implies (with an
informative message) when it detects that
{{SSLTestConfig.assumeSslIsSafeToTest()}} throws an assumption in the the
current JVM.
I considered just replacing {{SSLTestConfig.assumeSslIsSafeToTest()}} with a
{{SSLTestConfig.failTheBuildUnlesseSslIsSafeToTest()}} but realized that the
potential deluge of thousands of test failures that might occur for an aspiring
contributor who attempts to run Solr tests w/no idea their JDK is broken could
be overwhelming and scare people off before they even begin. A single clear
cut error (in addition to thousands of tests being {{SKIP}} ed) seemed more
inviting.
I should note: It's possible that down the road we will again find ourselves in
this situation...
bq. ...particularly once we realized that (at that time) every available
version of Java 13 was affected by SSL bugs...
...with some future "Java XX", whose every available 'ea' build we recognize as
being completely broken for SSL -- but we want still want to let jenkins try to
look for _other_ bugs w/o the "noise" of this test failing every build. If
that day comes, we can update {{SSLTestConfig.assumeSslIsSafeToTest()}} to
{{SKIP}} SSL on those JVM builds, and "whitelist" them in
{{TestSSLTestConfig.testFailIfUserRunsTestsWithJVMThatHasKnownSSLBugs}}.
> 'ant test' should fail on JVM's w/known SSL bugs
> ------------------------------------------------
>
> Key: SOLR-13747
> URL: https://issues.apache.org/jira/browse/SOLR-13747
> Project: Solr
> Issue Type: Test
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Hoss Man
> Priority: Major
> Attachments: SOLR-13747.patch
>
>
> If {{ant test}} (or the future gradle equivalent) is run w/a JVM that has
> known SSL bugs, there should be an obvious {{BUILD FAILED}} because of this
> -- so the user knows they should upgrade their JVM (rather then relying on
> the user to notice that SSL tests were {{SKIP}} ed)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
