I don’t know whether we had some temporary glitch that broke lots of tests and 
they’ve been fixed or we had a major regression, but this needs to be addressed 
ASAP if they’re still failing. See everything below the line "ALL OF THE TESTS 
BELOW HERE HAVE ONLY FAILED IN THE LAST WEEK!” in this e-mail. I’ll raise a 
JIRA if we can’t get some traction quickly here.

Hey, stuff happens. there’s no problem with tests going totally weird for a 
while. If you can say “Oh, yeah, all those failures for class XYZ are probably 
fixed” that’s fine.

Gosh-a-rooni, I hope my logging changes aren’t the culprit (gulp)….

Hoss’ rolllup for the last 24 hours is not encouraging in terms of the
problem already being fixed. There are lots of failures in some
classes, notably:

CloudHttp2SolrClientTest
CollectionsAPIDistributedZkTest
DeleteReplicaTest
TestDocCollectionWatcher

Unfortunately, the failure rate is not very high so reliably 
reproducing is hard.

I’ve reproduced the last week’s failure in this e-mail, full 
report attached. 

Here’s Hoss’ rollup:
http://fucit.org/solr-jenkins-reports/failure-report.html

Usual synopsis:

Raw fail count by week totals, most recent week first (corresponds to bits):
Week: 0  had  343 failures
Week: 1  had  86 failures
Week: 2  had  78 failures
Week: 3  had  117 failures


********Failures in Hoss' reports for the last 4 rollups.

There were 497 unannotated tests that failed in Hoss' rollups. Ordered by the 
date I downloaded the rollup file, newest->oldest. See above for the dates the 
files were collected 
These tests were NOT BadApple'd or AwaitsFix’d

Failures in the last 4 reports..
   Report   Pct     runs    fails           test
     0123   0.7     1617     11      
ConnectionManagerTest.testReconnectWhenZkDisappeared
     0123   1.5     1606     12      ExecutePlanActionTest.testTaskTimeout
     0123   1.6     1320     19      MultiThreadedOCPTest.test
     0123   1.0     1620     13      RollingRestartTest.test
     0123   1.2     1617     12      SearchRateTriggerTest.testWaitForElapsed
     0123   3.8      119      7      ShardSplitTest.testSplitWithChaosMonkey
     0123   0.3     1519      7      TestInPlaceUpdatesDistrib.test
     0123   0.7     1629     14      
TestIndexWriterDelete.testDeleteAllNoDeadLock
     0123   2.4     1548     18      TestPackages.testPluginLoading
     0123   0.3     1587      4      UnloadDistributedZkTest.test
********************************************

FAILURES IN THE LAST WEEK (343!)
Look particularly at the ones with only a zero in the “Report” column, those are
failures that were _not_ in the previous 3 week’s rollups.

   Report   Pct     runs    fails           test
     012    0.5     1165      4      CustomHighlightComponentTest.test
     012    1.0     1168      6      
NodeMarkersRegistrationTest.testNodeMarkersRegistration
     012    1.0     1170      8      TestCryptoKeys.test
     01 3   0.7     1233     11      LeaderFailoverAfterPartitionTest.test
     01 3  63.2      102     39      StressHdfsTest.test
     01     0.3      709      2      
ScheduledTriggerIntegrationTest.testScheduledTrigger
     01     0.2      768      2      ShardRoutingTest.test
     01     2.6      807     22      TestAllFilesHaveChecksumFooter.test
     01     2.6      808     22      TestAllFilesHaveCodecHeader.test
     01     0.2      769      2      TestCloudSchemaless.test
     01     0.2      769      2      TestDynamicLoading.testDynamicLoading
     01     0.3      707      2      TestDynamicLoadingUrl.testDynamicLoadingUrl
     01     0.5      767      4      TestPointFields.testFloatPointStats
     01    27.1       83     19      TestSQLHandler.doTest
     01     0.2      794     12      TestSameScoresWithThreads.test
     01     2.6      806     22      TestShardSearching.testSimple
     01     0.5      726      4      TestSimScenario.testSplitShard
     01     1.1      726      7      TestSimScenario.testSuggestions
     01     0.3      771      2      TestWithCollection.testAddReplicaSimple
     0 23   0.3     1223      4      
CdcrVersionReplicationTest.testCdcrDocVersions
     0 23   0.8     1172      6      
CloudHttp2SolrClientTest.testRetryUpdatesWhenClusterStateIsStale
     0 23   1.4     1202      8      CollectionsAPISolrJTest.testColStatus
     0 23   1.0     1249     11      HttpPartitionTest.test
     0 23   1.1     1210      8      HttpPartitionWithTlogReplicasTest.test
     0 23   0.5     1258      4      ShardSplitTest.testSplitShardWithRuleLink
     0 23   0.2     1231      4      
TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast
     0 23   0.2     1232      6      TestSolrConfigHandlerCloud.test
     0 2    0.3      767      2      
DocValuesNotIndexedTest.testGroupingDVOnlySortLast
     0 2    0.3      750      2      TestLBHttp2SolrClient.testTwoServers
     0 2    0.3      794      2      TestSolrCloudSnapshots.testSnapshots
     0 2   40.7       51     12      
TestXYMultiPolygonShapeQueries.testRandomBig
     0  3   0.2      842      2      ConnectionManagerTest.testLikelyExpired
     0  3   0.2      835      3      
LeaderElectionIntegrationTest.testSimpleSliceLeaderElection
     0  3   0.3      832      3      ReplaceNodeTest.test
     0  3   0.3      826      2      
SolrCloudReportersTest.testExplicitConfiguration
     0  3   0.2      833      2      
TestPullReplicaErrorHandling.testCantConnectToLeader
     0  3   2.4      867     21      TestRandomChains.testRandomChains
     0  3   0.5      830      3      
TestWithCollection.testMoveReplicaMainCollection

ALL OF THE TESTS BELOW HERE HAVE ONLY FAILED IN THE LAST WEEK!

     0      0.3      361      1      
AutoAddReplicasIntegrationTest.testClusterPropOverridesCollecitonProp
     0      2.9      380     11      BasicDistributedZkTest.test
     0      0.2      404      1      BasicZkTest.testBasic
     0      0.2      414      1      
ChaosMonkeySafeLeaderWithPullReplicasTest.test
     0     62.8      853    536      CheckHdfsIndexTest.doTest
     0      0.8      355      3      
CloudHttp2SolrClientTest.checkCollectionParameters
     0      0.8      355      3      
CloudHttp2SolrClientTest.customHttpClientTest
     0      1.1      355      4      
CloudHttp2SolrClientTest.preferLocalShardsTest
     0      0.8      355      3      
CloudHttp2SolrClientTest.preferReplicaTypesTest
     0      0.8      355      3      
CloudHttp2SolrClientTest.singleShardedPreferenceRules
     0      0.8      355      3      
CloudHttp2SolrClientTest.stateVersionParamTest
     0      1.1      355      4      CloudHttp2SolrClientTest.testAliasHandling
     0      0.8      355      3      
CloudHttp2SolrClientTest.testCollectionDoesntExist
     0      0.8      355      3      
CloudHttp2SolrClientTest.testInitializationWithSolrUrls
     0      0.8      355      3      
CloudHttp2SolrClientTest.testNonRetryableRequests
     0      1.1      355      4      
CloudHttp2SolrClientTest.testOverwriteOption
     0      0.8      355      3      
CloudHttp2SolrClientTest.testParallelUpdateQTime
     0      0.6      355      2      CloudHttp2SolrClientTest.testRouting
     0      0.8      355      3      CloudHttp2SolrClientTest.testShutdown
     0      0.6      355      2      
CloudHttp2SolrClientTest.testVersionsAreReturned
     0      0.8      355      3      
CloudHttp2SolrClientTest.testWrongZkChrootTest
     0      0.5      392      2      
CloudSolrClientTest.checkCollectionParameters
     0      0.3      392      1      CloudSolrClientTest.preferLocalShardsTest
     0      0.3      392      1      CloudSolrClientTest.preferReplicaTypesTest
     0      0.3      357      1      
CloudSolrClientTest.singleShardedPreferenceRules
     0      0.5      392      2      CloudSolrClientTest.stateVersionParamTest
     0      0.5      392      2      CloudSolrClientTest.testAliasHandling
     0      0.3      392      1      
CloudSolrClientTest.testInitializationWithSolrUrls
     0      0.3      392      1      
CloudSolrClientTest.testNonRetryableRequests
     0      0.3      392      1      CloudSolrClientTest.testOverwriteOption
     0      0.3      392      1      CloudSolrClientTest.testParallelUpdateQTime
     0      0.3      357      1      CloudSolrClientTest.testPing
     0      0.5      392      2      
CloudSolrClientTest.testRetryUpdatesWhenClusterStateIsStale
     0      0.3      392      1      CloudSolrClientTest.testShutdown
     0      0.5      392      2      CloudSolrClientTest.testVersionsAreReturned
     0      0.3      392      1      CloudSolrClientTest.testWrongZkChrootTest
     0      0.3      399      1      
CollectionsAPIAsyncDistributedZkTest.testSolrJAPICalls
     0      1.3      397      5      
CollectionsAPIDistributedZkTest.addReplicaTest
     0      0.8      397      3      
CollectionsAPIDistributedZkTest.deleteCollectionOnlyInZk
     0      1.0      397      4      
CollectionsAPIDistributedZkTest.deleteCollectionRemovesStaleZkCollectionsNode
     0      0.8      397      3      
CollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection
     0      0.3      397      1      
CollectionsAPIDistributedZkTest.testBadActionNames
     0      0.8      397      3      
CollectionsAPIDistributedZkTest.testCollectionReload
     0      1.0      397      4      
CollectionsAPIDistributedZkTest.testCollectionsAPI
     0      1.3      397      5      
CollectionsAPIDistributedZkTest.testCoresAreDistributedAcrossNodes
     0      1.0      397      4      
CollectionsAPIDistributedZkTest.testCreateNodeSet
     0      1.3      397      5      
CollectionsAPIDistributedZkTest.testCreateShouldFailOnExistingCore
     0      0.3      397      1      
CollectionsAPIDistributedZkTest.testCreationAndDeletion
     0      0.5      397      2      
CollectionsAPIDistributedZkTest.testDeleteNonExistentCollection
     0      0.8      397      3      
CollectionsAPIDistributedZkTest.testMaxNodesPerShard
     0      0.8      397      3      
CollectionsAPIDistributedZkTest.testMissingNumShards
     0      0.5      397      2      
CollectionsAPIDistributedZkTest.testMissingRequiredParameters
     0      0.8      397      3      
CollectionsAPIDistributedZkTest.testNoConfigSetExist
     0      0.8      397      3      
CollectionsAPIDistributedZkTest.testSpecificConfigsets
     0      0.8      397      3      
CollectionsAPIDistributedZkTest.testTooManyReplicas
     0      1.0      397      4      
CollectionsAPIDistributedZkTest.testZeroNumShards
     0      0.7      405      3      
CollectionsAPISolrJTest.testAddAndDeleteReplica
     0      0.2      405      1      
CollectionsAPISolrJTest.testAddAndDeleteReplicaProp
     0      0.7      405      3      
CollectionsAPISolrJTest.testBalanceShardUnique
     0      0.5      405      2      
CollectionsAPISolrJTest.testCloudInfoInCoreStatus
     0      1.5      405      6      CollectionsAPISolrJTest.testClusterProp
     0      0.5      405      2      CollectionsAPISolrJTest.testCollectionProp
     0      0.5      405      2      
CollectionsAPISolrJTest.testCreateAndDeleteAlias
     0      0.5      405      2      
CollectionsAPISolrJTest.testCreateAndDeleteCollection
     0      0.7      405      3      
CollectionsAPISolrJTest.testCreateCollWithDefaultClusterPropertiesNewFormat
     0      1.7      405      7      
CollectionsAPISolrJTest.testCreateCollWithDefaultClusterPropertiesOldFormat
     0      0.7      405      3      
CollectionsAPISolrJTest.testCreateCollectionWithPropertyParam
     0      1.5      405      6      
CollectionsAPISolrJTest.testCreateWithDefaultConfigSet
     0      1.9      370      7      
CollectionsAPISolrJTest.testDeleteAliasedCollection
     0      1.2      405      5      CollectionsAPISolrJTest.testList
     0      1.0      405      4      
CollectionsAPISolrJTest.testModifyCollectionAttribute
     0      1.2      405      5      CollectionsAPISolrJTest.testOverseerStatus
     0      0.8      370      3      
CollectionsAPISolrJTest.testReadOnlyCollection
     0      1.6      370      6      
CollectionsAPISolrJTest.testRenameCollection
     0      0.7      405      3      CollectionsAPISolrJTest.testSplitShard
     0      0.5      366      2      ComputePlanActionTest.testNodeAdded
     0      1.1      376      4      DeleteReplicaTest.deleteLiveReplicaTest
     0      1.5      411      6      
DeleteReplicaTest.deleteReplicaAndVerifyDirectoryCleanup
     0      0.5      411      2      DeleteReplicaTest.deleteReplicaByCount
     0      0.5      376      2      
DeleteReplicaTest.deleteReplicaByCountForAllShards
     0      1.2      411      5      
DeleteReplicaTest.deleteReplicaFromClusterState
     0      1.5      411      6      
DeleteReplicaTest.deleteReplicaFromClusterStateLegacy
     0      1.7      411      7      DeleteReplicaTest.deleteReplicaOnIndexing
     0      0.8      376      3      
DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica
     0      1.3      376      5      
DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplicaLegacy
     0      0.7      405      3      DeleteShardTest.test
     0      0.2      405      1      
DeleteShardTest.testDirectoryCleanupAfterDeleteShard
     0      0.5      366      2      
DistribDocExpirationUpdateProcessorTest.testBasicAuth
     0      0.5      366      2      
DistribDocExpirationUpdateProcessorTest.testNoAuth
     0      0.2      402      1      DistributedQueueTest.testDistributedQueue
     0      0.7      402      3      
DistributedQueueTest.testDistributedQueueBlocking
     0      0.7      402      3      
DistributedQueueTest.testDistributedQueueCache
     0      0.2      402      1      DistributedQueueTest.testLeakChildWatcher
     0      0.5      402      2      DistributedQueueTest.testLocallyOffer
     0      0.7      402      3      DistributedQueueTest.testPeekElements
     0      0.2      403      1      ExecutePlanActionTest.testExecute
     0      0.5      403      2      ExecutePlanActionTest.testIntegration
     0      0.7      403      3      ExecutePlanActionTest.testTaskFail
     0      0.3      370      1      
IndexSizeTriggerSizeEstimationTest.testEstimatedIndexSize
     0      0.3      366      1      IndexSizeTriggerTest.testInvalidSplitConfig
     0      0.3      366      1      IndexSizeTriggerTest.testMergeIntegration
     0      0.3      366      1      IndexSizeTriggerTest.testSplitConfig
     0      0.3      366      1      IndexSizeTriggerTest.testSplitIntegration
     0      0.3      366      1      IndexSizeTriggerTest.testTrigger
     0      0.3      366      1      IndexSizeTriggerTest.testValidProperties
     0      0.3      360      1      
JWTAuthPluginIntegrationTest.infoRequestWithoutToken
     0      0.3      360      1      JWTAuthPluginIntegrationTest.testMetrics
     0      0.2      403      1      LeaderElectionTest.testBasic
     0      0.5      403      2      LeaderElectionTest.testCancelElection
     0      0.5      403      2      LeaderElectionTest.testElection
     0      0.5      403      2      LeaderElectionTest.testParallelElection
     0      0.5      403      2      LeaderElectionTest.testStressElection
     0      0.2      404      1      LeaderFailureAfterFreshStartTest.test
     0      0.3      367      1      MoveReplicaTest.testFailedMove
     0      0.3      362      1      NestedShardedAtomicUpdateTest.test
     0      0.5      400      2      
NodeAddedTriggerIntegrationTest.testNodeAddedTriggerRestoreState
     0      0.3      398      1      NodeAddedTriggerTest.testListenerAcceptance
     0      0.3      398      1      NodeAddedTriggerTest.testRestoreState
     0      0.3      398      1      NodeAddedTriggerTest.testTrigger
     0      0.3      397      1      
NodeLostTriggerIntegrationTest.testNodeLostTriggerRestoreState
     0      0.3      398      1      
OutOfBoxZkACLAndCredentialsProvidersTest.testOpenACLUnsafeAllover
     0      0.3      398      1      
OutOfBoxZkACLAndCredentialsProvidersTest.testOutOfBoxSolrZkClient
     0      0.3      400      1      
OverriddenZkACLAndCredentialsProvidersTest.testAllCredentialsSolrZkClientFactoryUsingCompletelyNewProviders
     0      0.3      400      1      
OverriddenZkACLAndCredentialsProvidersTest.testAllCredentialsSolrZkClientFactoryUsingVMParamsProvidersButWithDifferentVMParamsNames
     0      0.3      400      1      
OverriddenZkACLAndCredentialsProvidersTest.testNoCredentialsSolrZkClientFactoryUsingCompletelyNewProviders
     0      0.3      400      1      
OverriddenZkACLAndCredentialsProvidersTest.testReadonlyCredentialsSolrZkClientFactoryUsingCompletelyNewProviders
     0      0.3      400      1      
OverriddenZkACLAndCredentialsProvidersTest.testReadonlyCredentialsSolrZkClientFactoryUsingVMParamsProvidersButWithDifferentVMParamsNames
     0      0.3      400      1      
OverriddenZkACLAndCredentialsProvidersTest.testWrongCredentialsSolrZkClientFactoryUsingCompletelyNewProviders
     0      0.3      400      1      
OverriddenZkACLAndCredentialsProvidersTest.testWrongCredentialsSolrZkClientFactoryUsingVMParamsProvidersButWithDifferentVMParamsNames
     0      0.3      400      1      
OverseerTaskQueueTest.testDistributedQueueCache
     0      0.3      400      1      OverseerTaskQueueTest.testLocallyOffer
     0      0.3      400      1      
PingRequestHandlerTest.testPingInClusterWithNoHealthCheck
     0      0.5      398      2      RecoveryAfterSoftCommitTest.test
     0      0.5      396      2      ReplicationFactorTest.test
     0      0.7      412      3      ShardSplitTest.testSplitLocking
     0      0.8      377      3      ShardSplitTest.testSplitMixedReplicaTypes
     0      0.5      411      2      
ShardSplitTest.testSplitStaticIndexReplication
     0      0.3      353      1      SolrZkClientTest.getConfig
     0      0.3      353      1      SolrZkClientTest.testWrappingWatches
     0      0.3      398      1      SpatialHeatmapFacetsTest.testClassicFacets
     0      0.3      398      1      SpatialHeatmapFacetsTest.testJsonFacets
     0      0.3     1194      3      SpatialHeatmapFacetsTest.testPng
     0      1.0      209      2      
StreamDecoratorTest.testParallelTerminatingDaemonUpdateStream
     0      0.3      397      1      SystemLogListenerTest.test
     0      0.5      388      2      
TestCloudCollectionsListeners.testCollectionDeletion
     0      0.3      388      1      
TestCloudCollectionsListeners.testSimpleCloudCollectionsListener
     0      0.3      397      1      
TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeaderAfterRestart
     0      0.3      396      1      
TestCloudSearcherWarming.testPeersyncFailureReplicationSuccess
     0      0.8      354      3      
TestCollectionStateWatchers.testCanWaitForNonexistantCollection
     0      0.8      354      3      
TestCollectionStateWatchers.testCollectionWatchWithShutdownOfActiveNode
     0      0.8      354      3      
TestCollectionStateWatchers.testCollectionWatchWithShutdownOfUnusedNode
     0      0.8      389      3      
TestCollectionStateWatchers.testDeletionsTriggerWatches
     0      0.6      354      2      
TestCollectionStateWatchers.testLiveNodeChangesTriggerWatches
     0      0.8      389      3      
TestCollectionStateWatchers.testPredicateFailureTimesOut
     0      0.8      389      3      
TestCollectionStateWatchers.testStateWatcherChecksCurrentStateOnRegister
     0      0.8      389      3      
TestCollectionStateWatchers.testWaitForStateChecksCurrentState
     0      0.8      389      3      
TestCollectionStateWatchers.testWaitForStateWatcherIsRetainedOnPredicateFailure
     0      0.5      389      2      
TestCollectionStateWatchers.testWatcherIsRemovedAfterTimeout
     0      0.5      389      2      
TestCollectionStateWatchers.testWatchesWorkForStateFormat1
     0      0.3      400      1      
TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete
     0      0.3      399      1      TestConfigSetsAPI.testCreate
     0      0.3      399      1      TestConfigSetsAPI.testList
     0      0.3      399      1      TestConfigSetsAPI.testUpload
     0      0.3      399      1      TestConfigSetsAPI.testUploadDisabled
     0      0.3      399      1      TestConfigSetsAPI.testUploadErrors
     0      0.3      399      1      
TestConfigSetsAPI.testUploadWithScriptUpdateProcessor
     0      0.3      399      1      
TestConfigSetsAPI.testUserAndTestDefaultConfigsetsAreSame
     0      0.3      400      1      TestDistribDocBasedVersion.test
     0      0.3      354      1      
TestDocCollectionWatcher.testCanWaitForNonexistantCollection
     0      0.6      354      2      
TestDocCollectionWatcher.testDeletionsTriggerWatches
     0      0.6      354      2      
TestDocCollectionWatcher.testPredicateFailureTimesOut
     0      0.6      354      2      
TestDocCollectionWatcher.testStateWatcherChecksCurrentStateOnRegister
     0      0.6      354      2      
TestDocCollectionWatcher.testWaitForStateChecksCurrentState
     0      0.6      354      2      
TestDocCollectionWatcher.testWaitForStateWatcherIsRetainedOnPredicateFailure
     0      0.6      354      2      
TestDocCollectionWatcher.testWatcherIsRemovedAfterTimeout
     0      0.6      354      2      
TestDocCollectionWatcher.testWatchesWorkForStateFormat1
     0     14.3       21      3      
TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit
     0      0.2      414      1      TestDuelingCodecs.testCrazyReaderEquals
     0      4.5       22      1      
TestDuelingCodecsAtNight.testCrazyReaderEquals
     0      0.2      419      1      
TestIndexWriterWithThreads.testIOExceptionDuringWriteSegmentWithThreads
     0      0.3      352      1      TestJaegerConfigurator.testInjected
     0      0.3      387      1      TestLTROnSolrCloud.testSimpleQuery
     0      1.2      402      5      TestPointFields.testDoublePointStats
     0      0.2      402      1      
TestPointFields.testIntPointFieldMultiValuedRangeFacet
     0      1.0      406      4      TestRandomDVFaceting.testRandomFaceting
     0      0.5      380      2      TestSimScenario.testAutoAddReplicas
     0      0.3      397      1      TestSimTriggerIntegration.testCooldown
     0      0.2      419      1      
TestSimpleExplanationsWithFillerDocs.testMultiFieldBQofPQ5
     0      0.2      402      1      TestSolrConfigHandlerConcurrent.test
     0      0.5      366      2      TestWithCollection.testAddReplicaWithPolicy
     0      0.3      400      1      TestWithCollection.testCreateCollection
     0      0.3      400      1      
TestWithCollection.testCreateCollectionNoWithCollection
     0      0.3      400      1      TestWithCollection.testDeleteWithCollection
     0      0.3      400      1      
TestWithCollection.testMoveReplicaWithCollection
     0      0.3      366      1      TestWithCollection.testNodeAdded
     0      0.2      401      1      ZkCLITest.testInvalidZKAddress
     0      0.2      401      1      ZkCLITest.testPutFile
     0      0.3      394      1      ZkSolrClientTest.testMakeRootNode
     0      0.3      394      1      ZkSolrClientTest.testReconnect
     0      0.3      394      1      ZkSolrClientTest.testWatchChildren




DO NOT ENABLE LIST:
    MoveReplicaHDFSTest.testFailedMove
    MoveReplicaHDFSTest.testNormalFailedMove
    TestControlledRealTimeReopenThread.testCRTReopen
    TestICUNormalizer2CharFilter.testRandomStrings
    TestICUTokenizerCJK
    TestImpersonationWithHadoopAuth.testForwarding
    TestLTRReRankingPipeline.testDifferentTopN
    TestRandomChains


DO NOT ANNOTATE LIST
    CdcrBidirectionalTest.testBiDir
    IndexSizeTriggerTest.testMergeIntegration
    IndexSizeTriggerTest.testMixedBounds
    IndexSizeTriggerTest.testSplitIntegration
    IndexSizeTriggerTest.testTrigger
    InfixSuggestersTest.testShutdownDuringBuild
    ShardSplitTest.test
    ShardSplitTest.testSplitMixedReplicaTypes
    ShardSplitTest.testSplitWithChaosMonkey
    Test2BPostings.test
    TestLatLonShapeQueries.testRandomBig
    TestPackedInts.testPackedLongValues
    TestRandomChains.testRandomChainsWithLargeStrings
    TestTriggerIntegration.testSearchRate

Processing file (History bit 3): HOSS-2020-05-04.csv
Processing file (History bit 2): HOSS-2020-04-27.csv
Processing file (History bit 1): HOSS-2020-04-20.csv
Processing file (History bit 0): HOSS-2020-04-13.csv


Number of AwaitsFix: 42 Number of BadApples: 4


**Annotated tests that didn't fail in the last 4 weeks.

  **Tests removed from the next two lists because they were specified in 
'doNotEnable' in the properties file
     MoveReplicaHDFSTest.testNormalFailedMove

  **Annotations can be removed from the following tests because they haven't 
failed in the last 4 rollups.

  **Methods: 0


Raw fail count by week totals, most recent week first (corresponds to bits):
Week: 0  had  343 failures
Week: 1  had  86 failures
Week: 2  had  78 failures
Week: 3  had  117 failures


** ******Failures in Hoss' reports for the last 4 rollups.

There were 497 unannotated tests that failed in Hoss' rollups. Ordered by the 
date I downloaded the rollup file, newest->oldest. See above for the dates the 
files were collected 
These tests were NOT BadApple'd or AwaitsFix'd

Failures in the last 4 reports..
   Report   Pct     runs    fails           test
     0123   0.7     1617     11      
ConnectionManagerTest.testReconnectWhenZkDisappeared
     0123   1.5     1606     12      ExecutePlanActionTest.testTaskTimeout
     0123   1.6     1320     19      MultiThreadedOCPTest.test
     0123   1.0     1620     13      RollingRestartTest.test
     0123   1.2     1617     12      SearchRateTriggerTest.testWaitForElapsed
     0123   3.8      119      7      ShardSplitTest.testSplitWithChaosMonkey
     0123   0.3     1519      7      TestInPlaceUpdatesDistrib.test
     0123   0.7     1629     14      
TestIndexWriterDelete.testDeleteAllNoDeadLock
     0123   2.4     1548     18      TestPackages.testPluginLoading
     0123   0.3     1587      4      UnloadDistributedZkTest.test
********************************************

Failures over the last 4 weeks, but not every week. Ordered most-recent first:



     012    0.5     1165      4      CustomHighlightComponentTest.test
     012    1.0     1168      6      
NodeMarkersRegistrationTest.testNodeMarkersRegistration
     012    1.0     1170      8      TestCryptoKeys.test
     01 3   0.7     1233     11      LeaderFailoverAfterPartitionTest.test
     01 3  63.2      102     39      StressHdfsTest.test
     01     0.3      709      2      
ScheduledTriggerIntegrationTest.testScheduledTrigger
     01     0.2      768      2      ShardRoutingTest.test
     01     2.6      807     22      TestAllFilesHaveChecksumFooter.test
     01     2.6      808     22      TestAllFilesHaveCodecHeader.test
     01     0.2      769      2      TestCloudSchemaless.test
     01     0.2      769      2      TestDynamicLoading.testDynamicLoading
     01     0.3      707      2      TestDynamicLoadingUrl.testDynamicLoadingUrl
     01     0.5      767      4      TestPointFields.testFloatPointStats
     01    27.1       83     19      TestSQLHandler.doTest
     01     0.2      794     12      TestSameScoresWithThreads.test
     01     2.6      806     22      TestShardSearching.testSimple
     01     0.5      726      4      TestSimScenario.testSplitShard
     01     1.1      726      7      TestSimScenario.testSuggestions
     01     0.3      771      2      TestWithCollection.testAddReplicaSimple
     0 23   0.3     1223      4      
CdcrVersionReplicationTest.testCdcrDocVersions
     0 23   0.8     1172      6      
CloudHttp2SolrClientTest.testRetryUpdatesWhenClusterStateIsStale
     0 23   1.4     1202      8      CollectionsAPISolrJTest.testColStatus
     0 23   1.0     1249     11      HttpPartitionTest.test
     0 23   1.1     1210      8      HttpPartitionWithTlogReplicasTest.test
     0 23   0.5     1258      4      ShardSplitTest.testSplitShardWithRuleLink
     0 23   0.2     1231      4      
TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast
     0 23   0.2     1232      6      TestSolrConfigHandlerCloud.test
     0 2    0.3      767      2      
DocValuesNotIndexedTest.testGroupingDVOnlySortLast
     0 2    0.3      750      2      TestLBHttp2SolrClient.testTwoServers
     0 2    0.3      794      2      TestSolrCloudSnapshots.testSnapshots
     0 2   40.7       51     12      
TestXYMultiPolygonShapeQueries.testRandomBig
     0  3   0.2      842      2      ConnectionManagerTest.testLikelyExpired
     0  3   0.2      835      3      
LeaderElectionIntegrationTest.testSimpleSliceLeaderElection
     0  3   0.3      832      3      ReplaceNodeTest.test
     0  3   0.3      826      2      
SolrCloudReportersTest.testExplicitConfiguration
     0  3   0.2      833      2      
TestPullReplicaErrorHandling.testCantConnectToLeader
     0  3   2.4      867     21      TestRandomChains.testRandomChains
     0  3   0.5      830      3      
TestWithCollection.testMoveReplicaMainCollection
     0      0.3      361      1      
AutoAddReplicasIntegrationTest.testClusterPropOverridesCollecitonProp
     0      2.9      380     11      BasicDistributedZkTest.test
     0      0.2      404      1      BasicZkTest.testBasic
     0      0.2      414      1      
ChaosMonkeySafeLeaderWithPullReplicasTest.test
     0     62.8      853    536      CheckHdfsIndexTest.doTest
     0      0.8      355      3      
CloudHttp2SolrClientTest.checkCollectionParameters
     0      0.8      355      3      
CloudHttp2SolrClientTest.customHttpClientTest
     0      1.1      355      4      
CloudHttp2SolrClientTest.preferLocalShardsTest
     0      0.8      355      3      
CloudHttp2SolrClientTest.preferReplicaTypesTest
     0      0.8      355      3      
CloudHttp2SolrClientTest.singleShardedPreferenceRules
     0      0.8      355      3      
CloudHttp2SolrClientTest.stateVersionParamTest
     0      1.1      355      4      CloudHttp2SolrClientTest.testAliasHandling
     0      0.8      355      3      
CloudHttp2SolrClientTest.testCollectionDoesntExist
     0      0.8      355      3      
CloudHttp2SolrClientTest.testInitializationWithSolrUrls
     0      0.8      355      3      
CloudHttp2SolrClientTest.testNonRetryableRequests
     0      1.1      355      4      
CloudHttp2SolrClientTest.testOverwriteOption
     0      0.8      355      3      
CloudHttp2SolrClientTest.testParallelUpdateQTime
     0      0.6      355      2      CloudHttp2SolrClientTest.testRouting
     0      0.8      355      3      CloudHttp2SolrClientTest.testShutdown
     0      0.6      355      2      
CloudHttp2SolrClientTest.testVersionsAreReturned
     0      0.8      355      3      
CloudHttp2SolrClientTest.testWrongZkChrootTest
     0      0.5      392      2      
CloudSolrClientTest.checkCollectionParameters
     0      0.3      392      1      CloudSolrClientTest.preferLocalShardsTest
     0      0.3      392      1      CloudSolrClientTest.preferReplicaTypesTest
     0      0.3      357      1      
CloudSolrClientTest.singleShardedPreferenceRules
     0      0.5      392      2      CloudSolrClientTest.stateVersionParamTest
     0      0.5      392      2      CloudSolrClientTest.testAliasHandling
     0      0.3      392      1      
CloudSolrClientTest.testInitializationWithSolrUrls
     0      0.3      392      1      
CloudSolrClientTest.testNonRetryableRequests
     0      0.3      392      1      CloudSolrClientTest.testOverwriteOption
     0      0.3      392      1      CloudSolrClientTest.testParallelUpdateQTime
     0      0.3      357      1      CloudSolrClientTest.testPing
     0      0.5      392      2      
CloudSolrClientTest.testRetryUpdatesWhenClusterStateIsStale
     0      0.3      392      1      CloudSolrClientTest.testShutdown
     0      0.5      392      2      CloudSolrClientTest.testVersionsAreReturned
     0      0.3      392      1      CloudSolrClientTest.testWrongZkChrootTest
     0      0.3      399      1      
CollectionsAPIAsyncDistributedZkTest.testSolrJAPICalls
     0      1.3      397      5      
CollectionsAPIDistributedZkTest.addReplicaTest
     0      0.8      397      3      
CollectionsAPIDistributedZkTest.deleteCollectionOnlyInZk
     0      1.0      397      4      
CollectionsAPIDistributedZkTest.deleteCollectionRemovesStaleZkCollectionsNode
     0      0.8      397      3      
CollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection
     0      0.3      397      1      
CollectionsAPIDistributedZkTest.testBadActionNames
     0      0.8      397      3      
CollectionsAPIDistributedZkTest.testCollectionReload
     0      1.0      397      4      
CollectionsAPIDistributedZkTest.testCollectionsAPI
     0      1.3      397      5      
CollectionsAPIDistributedZkTest.testCoresAreDistributedAcrossNodes
     0      1.0      397      4      
CollectionsAPIDistributedZkTest.testCreateNodeSet
     0      1.3      397      5      
CollectionsAPIDistributedZkTest.testCreateShouldFailOnExistingCore
     0      0.3      397      1      
CollectionsAPIDistributedZkTest.testCreationAndDeletion
     0      0.5      397      2      
CollectionsAPIDistributedZkTest.testDeleteNonExistentCollection
     0      0.8      397      3      
CollectionsAPIDistributedZkTest.testMaxNodesPerShard
     0      0.8      397      3      
CollectionsAPIDistributedZkTest.testMissingNumShards
     0      0.5      397      2      
CollectionsAPIDistributedZkTest.testMissingRequiredParameters
     0      0.8      397      3      
CollectionsAPIDistributedZkTest.testNoConfigSetExist
     0      0.8      397      3      
CollectionsAPIDistributedZkTest.testSpecificConfigsets
     0      0.8      397      3      
CollectionsAPIDistributedZkTest.testTooManyReplicas
     0      1.0      397      4      
CollectionsAPIDistributedZkTest.testZeroNumShards
     0      0.7      405      3      
CollectionsAPISolrJTest.testAddAndDeleteReplica
     0      0.2      405      1      
CollectionsAPISolrJTest.testAddAndDeleteReplicaProp
     0      0.7      405      3      
CollectionsAPISolrJTest.testBalanceShardUnique
     0      0.5      405      2      
CollectionsAPISolrJTest.testCloudInfoInCoreStatus
     0      1.5      405      6      CollectionsAPISolrJTest.testClusterProp
     0      0.5      405      2      CollectionsAPISolrJTest.testCollectionProp
     0      0.5      405      2      
CollectionsAPISolrJTest.testCreateAndDeleteAlias
     0      0.5      405      2      
CollectionsAPISolrJTest.testCreateAndDeleteCollection
     0      0.7      405      3      
CollectionsAPISolrJTest.testCreateCollWithDefaultClusterPropertiesNewFormat
     0      1.7      405      7      
CollectionsAPISolrJTest.testCreateCollWithDefaultClusterPropertiesOldFormat
     0      0.7      405      3      
CollectionsAPISolrJTest.testCreateCollectionWithPropertyParam
     0      1.5      405      6      
CollectionsAPISolrJTest.testCreateWithDefaultConfigSet
     0      1.9      370      7      
CollectionsAPISolrJTest.testDeleteAliasedCollection
     0      1.2      405      5      CollectionsAPISolrJTest.testList
     0      1.0      405      4      
CollectionsAPISolrJTest.testModifyCollectionAttribute
     0      1.2      405      5      CollectionsAPISolrJTest.testOverseerStatus
     0      0.8      370      3      
CollectionsAPISolrJTest.testReadOnlyCollection
     0      1.6      370      6      
CollectionsAPISolrJTest.testRenameCollection
     0      0.7      405      3      CollectionsAPISolrJTest.testSplitShard
     0      0.5      366      2      ComputePlanActionTest.testNodeAdded
     0      1.1      376      4      DeleteReplicaTest.deleteLiveReplicaTest
     0      1.5      411      6      
DeleteReplicaTest.deleteReplicaAndVerifyDirectoryCleanup
     0      0.5      411      2      DeleteReplicaTest.deleteReplicaByCount
     0      0.5      376      2      
DeleteReplicaTest.deleteReplicaByCountForAllShards
     0      1.2      411      5      
DeleteReplicaTest.deleteReplicaFromClusterState
     0      1.5      411      6      
DeleteReplicaTest.deleteReplicaFromClusterStateLegacy
     0      1.7      411      7      DeleteReplicaTest.deleteReplicaOnIndexing
     0      0.8      376      3      
DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica
     0      1.3      376      5      
DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplicaLegacy
     0      0.7      405      3      DeleteShardTest.test
     0      0.2      405      1      
DeleteShardTest.testDirectoryCleanupAfterDeleteShard
     0      0.5      366      2      
DistribDocExpirationUpdateProcessorTest.testBasicAuth
     0      0.5      366      2      
DistribDocExpirationUpdateProcessorTest.testNoAuth
     0      0.2      402      1      DistributedQueueTest.testDistributedQueue
     0      0.7      402      3      
DistributedQueueTest.testDistributedQueueBlocking
     0      0.7      402      3      
DistributedQueueTest.testDistributedQueueCache
     0      0.2      402      1      DistributedQueueTest.testLeakChildWatcher
     0      0.5      402      2      DistributedQueueTest.testLocallyOffer
     0      0.7      402      3      DistributedQueueTest.testPeekElements
     0      0.2      403      1      ExecutePlanActionTest.testExecute
     0      0.5      403      2      ExecutePlanActionTest.testIntegration
     0      0.7      403      3      ExecutePlanActionTest.testTaskFail
     0      0.3      370      1      
IndexSizeTriggerSizeEstimationTest.testEstimatedIndexSize
     0      0.3      366      1      IndexSizeTriggerTest.testInvalidSplitConfig
     0      0.3      366      1      IndexSizeTriggerTest.testMergeIntegration
     0      0.3      366      1      IndexSizeTriggerTest.testSplitConfig
     0      0.3      366      1      IndexSizeTriggerTest.testSplitIntegration
     0      0.3      366      1      IndexSizeTriggerTest.testTrigger
     0      0.3      366      1      IndexSizeTriggerTest.testValidProperties
     0      0.3      360      1      
JWTAuthPluginIntegrationTest.infoRequestWithoutToken
     0      0.3      360      1      JWTAuthPluginIntegrationTest.testMetrics
     0      0.2      403      1      LeaderElectionTest.testBasic
     0      0.5      403      2      LeaderElectionTest.testCancelElection
     0      0.5      403      2      LeaderElectionTest.testElection
     0      0.5      403      2      LeaderElectionTest.testParallelElection
     0      0.5      403      2      LeaderElectionTest.testStressElection
     0      0.2      404      1      LeaderFailureAfterFreshStartTest.test
     0      0.3      367      1      MoveReplicaTest.testFailedMove
     0      0.3      362      1      NestedShardedAtomicUpdateTest.test
     0      0.5      400      2      
NodeAddedTriggerIntegrationTest.testNodeAddedTriggerRestoreState
     0      0.3      398      1      NodeAddedTriggerTest.testListenerAcceptance
     0      0.3      398      1      NodeAddedTriggerTest.testRestoreState
     0      0.3      398      1      NodeAddedTriggerTest.testTrigger
     0      0.3      397      1      
NodeLostTriggerIntegrationTest.testNodeLostTriggerRestoreState
     0      0.3      398      1      
OutOfBoxZkACLAndCredentialsProvidersTest.testOpenACLUnsafeAllover
     0      0.3      398      1      
OutOfBoxZkACLAndCredentialsProvidersTest.testOutOfBoxSolrZkClient
     0      0.3      400      1      
OverriddenZkACLAndCredentialsProvidersTest.testAllCredentialsSolrZkClientFactoryUsingCompletelyNewProviders
     0      0.3      400      1      
OverriddenZkACLAndCredentialsProvidersTest.testAllCredentialsSolrZkClientFactoryUsingVMParamsProvidersButWithDifferentVMParamsNames
     0      0.3      400      1      
OverriddenZkACLAndCredentialsProvidersTest.testNoCredentialsSolrZkClientFactoryUsingCompletelyNewProviders
     0      0.3      400      1      
OverriddenZkACLAndCredentialsProvidersTest.testReadonlyCredentialsSolrZkClientFactoryUsingCompletelyNewProviders
     0      0.3      400      1      
OverriddenZkACLAndCredentialsProvidersTest.testReadonlyCredentialsSolrZkClientFactoryUsingVMParamsProvidersButWithDifferentVMParamsNames
     0      0.3      400      1      
OverriddenZkACLAndCredentialsProvidersTest.testWrongCredentialsSolrZkClientFactoryUsingCompletelyNewProviders
     0      0.3      400      1      
OverriddenZkACLAndCredentialsProvidersTest.testWrongCredentialsSolrZkClientFactoryUsingVMParamsProvidersButWithDifferentVMParamsNames
     0      0.3      400      1      
OverseerTaskQueueTest.testDistributedQueueCache
     0      0.3      400      1      OverseerTaskQueueTest.testLocallyOffer
     0      0.3      400      1      
PingRequestHandlerTest.testPingInClusterWithNoHealthCheck
     0      0.5      398      2      RecoveryAfterSoftCommitTest.test
     0      0.5      396      2      ReplicationFactorTest.test
     0      0.7      412      3      ShardSplitTest.testSplitLocking
     0      0.8      377      3      ShardSplitTest.testSplitMixedReplicaTypes
     0      0.5      411      2      
ShardSplitTest.testSplitStaticIndexReplication
     0      0.3      353      1      SolrZkClientTest.getConfig
     0      0.3      353      1      SolrZkClientTest.testWrappingWatches
     0      0.3      398      1      SpatialHeatmapFacetsTest.testClassicFacets
     0      0.3      398      1      SpatialHeatmapFacetsTest.testJsonFacets
     0      0.3     1194      3      SpatialHeatmapFacetsTest.testPng
     0      1.0      209      2      
StreamDecoratorTest.testParallelTerminatingDaemonUpdateStream
     0      0.3      397      1      SystemLogListenerTest.test
     0      0.5      388      2      
TestCloudCollectionsListeners.testCollectionDeletion
     0      0.3      388      1      
TestCloudCollectionsListeners.testSimpleCloudCollectionsListener
     0      0.3      397      1      
TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeaderAfterRestart
     0      0.3      396      1      
TestCloudSearcherWarming.testPeersyncFailureReplicationSuccess
     0      0.8      354      3      
TestCollectionStateWatchers.testCanWaitForNonexistantCollection
     0      0.8      354      3      
TestCollectionStateWatchers.testCollectionWatchWithShutdownOfActiveNode
     0      0.8      354      3      
TestCollectionStateWatchers.testCollectionWatchWithShutdownOfUnusedNode
     0      0.8      389      3      
TestCollectionStateWatchers.testDeletionsTriggerWatches
     0      0.6      354      2      
TestCollectionStateWatchers.testLiveNodeChangesTriggerWatches
     0      0.8      389      3      
TestCollectionStateWatchers.testPredicateFailureTimesOut
     0      0.8      389      3      
TestCollectionStateWatchers.testStateWatcherChecksCurrentStateOnRegister
     0      0.8      389      3      
TestCollectionStateWatchers.testWaitForStateChecksCurrentState
     0      0.8      389      3      
TestCollectionStateWatchers.testWaitForStateWatcherIsRetainedOnPredicateFailure
     0      0.5      389      2      
TestCollectionStateWatchers.testWatcherIsRemovedAfterTimeout
     0      0.5      389      2      
TestCollectionStateWatchers.testWatchesWorkForStateFormat1
     0      0.3      400      1      
TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete
     0      0.3      399      1      TestConfigSetsAPI.testCreate
     0      0.3      399      1      TestConfigSetsAPI.testList
     0      0.3      399      1      TestConfigSetsAPI.testUpload
     0      0.3      399      1      TestConfigSetsAPI.testUploadDisabled
     0      0.3      399      1      TestConfigSetsAPI.testUploadErrors
     0      0.3      399      1      
TestConfigSetsAPI.testUploadWithScriptUpdateProcessor
     0      0.3      399      1      
TestConfigSetsAPI.testUserAndTestDefaultConfigsetsAreSame
     0      0.3      400      1      TestDistribDocBasedVersion.test
     0      0.3      354      1      
TestDocCollectionWatcher.testCanWaitForNonexistantCollection
     0      0.6      354      2      
TestDocCollectionWatcher.testDeletionsTriggerWatches
     0      0.6      354      2      
TestDocCollectionWatcher.testPredicateFailureTimesOut
     0      0.6      354      2      
TestDocCollectionWatcher.testStateWatcherChecksCurrentStateOnRegister
     0      0.6      354      2      
TestDocCollectionWatcher.testWaitForStateChecksCurrentState
     0      0.6      354      2      
TestDocCollectionWatcher.testWaitForStateWatcherIsRetainedOnPredicateFailure
     0      0.6      354      2      
TestDocCollectionWatcher.testWatcherIsRemovedAfterTimeout
     0      0.6      354      2      
TestDocCollectionWatcher.testWatchesWorkForStateFormat1
     0     14.3       21      3      
TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit
     0      0.2      414      1      TestDuelingCodecs.testCrazyReaderEquals
     0      4.5       22      1      
TestDuelingCodecsAtNight.testCrazyReaderEquals
     0      0.2      419      1      
TestIndexWriterWithThreads.testIOExceptionDuringWriteSegmentWithThreads
     0      0.3      352      1      TestJaegerConfigurator.testInjected
     0      0.3      387      1      TestLTROnSolrCloud.testSimpleQuery
     0      1.2      402      5      TestPointFields.testDoublePointStats
     0      0.2      402      1      
TestPointFields.testIntPointFieldMultiValuedRangeFacet
     0      1.0      406      4      TestRandomDVFaceting.testRandomFaceting
     0      0.5      380      2      TestSimScenario.testAutoAddReplicas
     0      0.3      397      1      TestSimTriggerIntegration.testCooldown
     0      0.2      419      1      
TestSimpleExplanationsWithFillerDocs.testMultiFieldBQofPQ5
     0      0.2      402      1      TestSolrConfigHandlerConcurrent.test
     0      0.5      366      2      TestWithCollection.testAddReplicaWithPolicy
     0      0.3      400      1      TestWithCollection.testCreateCollection
     0      0.3      400      1      
TestWithCollection.testCreateCollectionNoWithCollection
     0      0.3      400      1      TestWithCollection.testDeleteWithCollection
     0      0.3      400      1      
TestWithCollection.testMoveReplicaWithCollection
     0      0.3      366      1      TestWithCollection.testNodeAdded
     0      0.2      401      1      ZkCLITest.testInvalidZKAddress
     0      0.2      401      1      ZkCLITest.testPutFile
     0      0.3      394      1      ZkSolrClientTest.testMakeRootNode
     0      0.3      394      1      ZkSolrClientTest.testReconnect
     0      0.3      394      1      ZkSolrClientTest.testWatchChildren
      123   0.3     1196      5      
ComputePlanActionTest.testSelectedCollections
      123   0.3     1196      3      DaemonStreamApiTest.testAPIs
      123  80.0       21     15      SharedFSAutoReplicaFailoverTest.test
      123   0.3     1202     24      SystemCollectionCompatTest.testBackCompat
      12    4.0       50      2      
CdcrReplicationHandlerTest.testReplicationWithBufferedUpdates
      12    3.8       58      7      HdfsUnloadDistributedZkTest.test
      12    0.3      736      3      
LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud
      1 3   0.3      800      2      AutoScalingHandlerTest.testReadApi
      1 3   0.3      772      2      
AutoScalingHandlerTest.testSuggestionsWithPayload
      1     0.3      366      1      ActionThrottleTest.testBasics
      1     2.9      377     11      BJQParserTest.testToParentFilters
      1     0.3      366      1      ChaosMonkeyNothingIsSafeTest.test
      1     0.3      365      1      PeerSyncWithLeaderTest.test
      1     0.3      366      1      RulesTest.doIntegrationTest
      1     0.3      335      1      
StreamDecoratorTest.testParallelExecutorStream
      1     0.5      185      1      
StreamExpressionTest.testFeaturesSelectionStream
      1     1.9      371      7      TestBloomPostingsFormat.testInvertedWrite
      1     2.7      377     10      TestCheckIndex.testChecksumsOnly
      1     0.3      301      1      
TestDelegationWithHadoopAuth.testDelegationTokenRenew
      1     2.2      372      8      TestDirectPostingsFormat.testInvertedWrite
      1     1.6      370      6      TestFSTPostingsFormat.testInvertedWrite
      1     1.6      370      6      
TestFixedGapPostingsFormat.testInvertedWrite
      1     2.9      379     11      TestIndexWriterOutOfFileDescriptors.test
      1     0.3      367      1      TestOfflineSorter.testThreadSafety
      1     1.6      370      6      TestOrdsBlockTree.testInvertedWrite
      1    16.5      468     77      
TestPolicyCloud.testCreateCollectionAddShardUsingPolicy
      1     5.3      468     25      
TestPolicyCloud.testCreateCollectionSplitShard
      1     5.6      392     22      TestSearcherManager.testSearcherManager
      1     0.3      367      1      
TestSegmentSorting.testSegmentTerminateEarly
      1     0.3      336      1      
TestSimLargeCluster.testCreateLargeSimCollections
      1     0.3      336      1      TestSimLargeCluster.testFreediskTracking
      1     0.3      336      1      TestSimLargeCluster.testNodeLost
      1     0.3      364      1      TestSimLargeCluster.testSearchRate
      1    21.4       28      6      
TestSimpleTextPostingsFormat.testInvertedWrite
      1     0.3      365      1      
TestStressCloudBlindAtomicUpdates.test_dv_stored
      1     0.3      365      1      
TestStressCloudBlindAtomicUpdates.test_stored_idx
      1     1.9      371      7      
TestVarGapDocFreqIntervalPostingsFormat.testInvertedWrite
      1     0.3      366      1      
UninvertDocValuesMergePolicyTest.testIndexAndAddDocValues
       23   3.4      575     20      
Geo3dShapeWGS84ModelRectRelationTest.testGeoCircleRect
       23  23.8      106     48      HdfsSyncSliceTest.test
       23   0.3      839      9      SyncSliceTest.test
       23   0.3      831      4      TestCloudRecovery2.test
       23   1.0      843      8      
TestControlledRealTimeReopenThread.testControlledRealTimeReopenThread
       23   0.2      839      3      
TestReplicationHandler.doTestIndexAndConfigReplication
       23   0.1     1660      2      TestTlogReplica.testCreateDelete
       23   0.3      830      2      TestTlogReplica.testRemoveLeader
       2    0.3      400      1      
DocValuesNotIndexedTest.testGroupingDocAbsent
       2    6.9       29      2      
HdfsTlogReplayBufferedWhileIndexingTest.test
       2    0.4      273      1      
HealthCheckHandlerTest.testHealthCheckHandler
       2    0.5      400      2      
MetricTriggerIntegrationTest.testMetricTrigger
       2    0.3      398      1      NodeLostTriggerTest.testTrigger
       2    0.3      397      1      OverseerRolesTest.testOverseerRole
       2    3.8       26      1      RestartWhileUpdatingTest.test
       2    0.2      404      1      ShardSplitTest.testSplitShardWithRule
       2    0.3      400      1      TestAddIndexes.testAddIndexesWithRollback
       2    0.3      395      1      TestCloudJSONFacetSKG.testRandom
       2    0.5      401      2      TestCloudPivotFacet.test
       2    0.0     2396      1      TestJsonFacets.testPrelimSortingSingleNode
       2    0.3      395      1      TestLBHttp2SolrClient.testReliability
       2    0.3      395      1      TestLBHttp2SolrClient.testSimple
       2    0.3      392      1      TestLBHttpSolrClient.testReliability
       2    0.5      397      2      TestNamedUpdateProcessors.test
       2    0.0     3960      1      
TestSolr4Spatial2.testLLPDecodeIsStableAndPrecise
        3   0.2      433      1      
AliasIntegrationTest.testDeleteOneOfTwoCollectionsAliased
        3   3.7       27      1      HdfsBasicDistributedZkTest.test
        3   0.2      426      1      Http2SolrClientTest.testRedirect
        3   0.2      426      1      MathExpressionTest.testFuzzyKmeans
        3   0.2      432      1      
MetricsHistoryWithAuthIntegrationTest.testValuesAreCollected
        3   0.2      443      1      
ShardSplitTest.testSplitMixedReplicaTypesLink
        3   0.3      368      1      
SoftAutoCommitTest.testSoftAndHardCommitMaxTimeDelete
        3   9.4       32      3      Test2BPostings.test
        3   7.4       27      2      
TestAssertingPointsFormat.testRandomBinaryBig
        3   1.4      436      6      
TestAssertingPostingsFormat.testDocsAndFreqsAndPositionsAndOffsets
        3   1.4      442      6      
TestBlockPostingsFormat.testDocsAndFreqsAndPositions
        3   1.1      437      5      TestBlockPostingsFormat.testDocsOnly
        3   0.5      442      2      TestBlockPostingsFormat.testRandom
        3   1.2      433      5      TestBloomPostingsFormat.testRandom
        3   7.8      128     10      TestBufferedUpdates.testRamBytesUsed
        3   3.7       27      1      TestDuelingCodecsAtNight.testBigEquals
        3   0.7      431      3      TestFSTPostingsFormat.testRandom
        3   0.5      430      2      
TestFixedGapPostingsFormat.testDocsAndFreqsAndPositionsAndOffsetsAndPayloads
        3   0.2      433      1      TestIndexFileDeleter.testExcInDecRef
        3   0.2      432      1      
TestIndexWriterThreadsToSegments.testSegmentCountOnFlushRandom
        3   0.2      448      1      
TestIndexWriterWithThreads.testIOExceptionDuringAbortWithThreads
        3   0.9      448      4      
TestIndexWriterWithThreads.testRollbackAndCommitWithThreads
        3   0.2      431      1      TestLeaderElectionWithEmptyReplica.test
        3   3.7       27      1      
TestLucene80DocValuesFormat.testTermsEnumLongSharedPrefixes
        3   1.4      434      6      TestOrdsBlockTree.testRandom
        3   1.4      429      6      
TestRandomCollapseQParserPlugin.testRandomCollpaseWithSort
        3   0.2      437      1      
TestReplicationHandler.doTestIndexAndConfigAliasReplication
        3   0.2      429      1      
TestSTUniformSplitPostingFormat.testDocsAndFreqsAndPositionsAndOffsetsAndPayloads
        3   0.5      433      2      
TestSearcherManager.testConcurrentIndexCloseSearchAndRefresh
        3   0.7      430      3      TestSimLargeCluster.testAddNode
        3   0.2      428      1      TestSimNodeAddedTrigger.testTrigger
        3   0.2      428      1      
TestSolrCloudWithDelegationTokens.testDelegationTokenRenew
        3   0.5      428      2      
TestSolrDeletionPolicy1.testNumCommitsConfigured
        3   0.7      434      3      TestStressLiveNodes.testStress
        3   0.2      432      1      
TestUniformSplitPostingFormat.testDocsAndFreqsAndPositionsAndOffsetsAndPayloads
        3   0.9      432      4      TestUniformSplitPostingFormat.testRandom
        3   0.5      430      2      
TestVarGapDocFreqIntervalPostingsFormat.testDocsAndFreqsAndPositionsAndOffsetsAndPayloads
        3   0.9      432      4      
TestVarGapFixedIntervalPostingsFormat.testDocsAndFreqsAndPositionsAndOffsets
        3   1.2      433      5      
TestVarGapFixedIntervalPostingsFormat.testDocsAndFreqsAndPositionsAndOffsetsAndPayloads
        3   0.7      433      3      
TriggerSetPropertiesIntegrationTest.testSetProperties



***********************Annotations report

*****************AwaitsFix Annotations:


************Lucene AwaitsFix
TestControlledRealTimeReopenThread.testCRTReopen
   @AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/LUCENE-5737";)

TestICUNormalizer2CharFilter.testRandomStrings
   @AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/LUCENE-5595";)

TestICUTokenizerCJK.TestICUTokenizerCJK suite
   @AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/LUCENE-8222";)

TestMoreLikeThis.testMultiFieldShouldReturnPerFieldBooleanQuery
   @AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/LUCENE-7161";)


    ************Solr AwaitsFix
AutoAddReplicasIntegrationTest.testRapidStopStartStopWithPropChange
   @AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-13811";)

CdcrBidirectionalTest.testBiDir
   @AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/SOLR-12524";)

CdcrBootstrapTest.testBootstrapWithContinousIndexingOnSourceCluster
   @AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/SOLR-12028";)

CollectionsAPISolrJTest.testCreateAndDeleteShard
   @AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/SOLR-13021";)

ComputePlanActionTest.testNodeLost
   
@LuceneTestCase.AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";)

ConcurrentCreateRoutedAliasTest.ConcurrentCreateRoutedAliasTest suite
   @LuceneTestCase.AwaitsFix(bugUrl = 
"https://issues.apache.org/jira/browse/SOLR-12386";) // "Can't find resource"

LeaderTragicEventTest.LeaderTragicEventTest suite
   @AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-13237";)

MoveReplicaHDFSTest.MoveReplicaHDFSTest suite
   @AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-13924";)

MoveReplicaHDFSTest.testFailedMove
   @AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12080";) // 
added 03-Oct-2018

ReplaceNodeNoTargetTest.ReplaceNodeNoTargetTest suite
   @LuceneTestCase.AwaitsFix(bugUrl = 
"https://issues.apache.org/jira/browse/SOLR-11067";)

ReplaceNodeNoTargetTest.test
   @LuceneTestCase.AwaitsFix(bugUrl = 
"https://issues.apache.org/jira/browse/SOLR-11067";)

RoutedAliasUpdateProcessorTest.alias,
   
@org.apache.lucene.util.LuceneTestCase.AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-13696";)

SSLMigrationTest.SSLMigrationTest suite
   @AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";) // 
17-Mar-2018

SaslZkACLProviderTest.testSaslZkACLProvider
   @AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/SOLR-13075";)

ScheduledMaintenanceTriggerTest.testInactiveShardCleanup
   @AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";) // 
17-Mar-2018

ScheduledTriggerTest.testTrigger
   @AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";) // 
added 20-Sep-2018

SearchRateTriggerIntegrationTest.testDeleteNode
   @AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-13163";)

SearchRateTriggerTest.testTrigger
   @AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";)

SolrExporterIntegrationTest.SolrExporterIntegrationTest suite
   
@org.apache.lucene.util.LuceneTestCase.AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-13786";)

Tagger2Test.testStopWords
   @AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/LUCENE-8344";)

TestHierarchicalDocBuilder.testThreeLevelHierarchy
   @AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/SOLR-12801";) // 
this test fails easily under beasting

TestImpersonationWithHadoopAuth.testForwarding
   @AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/HADOOP-9893";)

TestJsonFacetRefinement.testProcessEmptyRefinement
   @AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12556";)

TestJsonFacetRefinement.testSortedSubFacetRefinementWhenParentOnlyReturnedByOneShardProcessEmpty
   @AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12556";)

TestLTRReRankingPipeline.testDifferentTopN
   @AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/SOLR-11134";)

TestMinMaxOnMultiValuedField.testDoubleFieldCache
   @AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/LUCENE-6709";)

TestMinMaxOnMultiValuedField.testFloatFieldCache
   @AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/LUCENE-6709";)

TestMinMaxOnMultiValuedField.testIntFieldCache
   @AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/LUCENE-6709";)

TestMinMaxOnMultiValuedField.testLongFieldCache
   @AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/LUCENE-6709";)

TestPullReplica.TestPullReplica suite
   @AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";)

TestSimComputePlanAction.testNodeLost
   @AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/SOLR-12028";) // 
if you beast this, eventually you will see

TestSimExtremeIndexing.TestSimExtremeIndexing suite
   
@org.apache.lucene.util.LuceneTestCase.AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12923";)

TestSimLargeCluster.testAddNode
   @AwaitsFix( bugUrl = "https://issues.apache.org/jira/browse/SOLR-14275";)

TestSimLargeCluster.testBasic
   @AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";) // 
this test hits a timeout easily

TestStressRecovery.TestStressRecovery suite
   
@LuceneTestCase.AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";)
 // 6-Sep-2018

TestTlogReplayVsRecovery.TestTlogReplayVsRecovery suite
   
@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-13486;https://issues.apache.org/jira/browse/SOLR-14183";)

TimeRoutedAliasUpdateProcessorTest.testDateMathInStart
   @AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-13943";)

TriLevelCompositeIdRoutingTest.test
   @AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-13369";)



*****************BadApple Annotations:

************Lucene BadApple

************Solr BadApple
MoveReplicaHDFSTest.testNormalFailedMove
   @BadApple(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";) // 
14-Oct-2018

SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
   
@LuceneTestCase.BadApple(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";)
 // added 20-Jul-2018

TimeRoutedAliasUpdateProcessorTest.TimeRoutedAliasUpdateProcessorTest suite
   @LuceneTestCase.BadApple(bugUrl = 
"https://issues.apache.org/jira/browse/SOLR-13059";)


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to