Short form:
Holding reasonably steady, except:
PackageManagerCLITest.testPackeageManager is failing over 50% of the time.
MoverReplicaHDFSTest.test failing over 40% of the time.
TestFstDirectAddressing.testDeDupeTails is failing 18% of the time.
Full output attached:
There were 154 unannotated tests that failed in Hoss' rollups. Ordered by the
date I downloaded the rollup file, newest->oldest. See above for the dates the
files were collected
These tests were NOT BadApple'd or AwaitsFix'd
All tests that failed 4 weeks running will be BadApple'd unless there are
objections
Failures in the last 4 reports..
Report Pct runs fails test
0123 3.3 917 45 BasicAuthIntegrationTest.testBasicAuth
0123 0.5 943 7
DimensionalRoutedAliasUpdateProcessorTest.testCatTime
0123 2.7 943 18
DimensionalRoutedAliasUpdateProcessorTest.testTimeCat
0123 8.3 961 91
LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud
0123 43.8 202 85 MoveReplicaHDFSTest.test
0123 0.5 903 5
ReindexCollectionTest.testSameTargetReindexing
0123 4.8 104 5 ShardSplitTest.testSplitWithChaosMonkey
0123 0.5 918 9 SystemCollectionCompatTest.testBackCompat
0123 4.8 941 52
TestCloudSearcherWarming.testRepFactor1LeaderStartup
0123 2.7 946 77
TestModelManagerPersistence.testFilePersistence
0123 2.2 940 70
TestModelManagerPersistence.testWrapperModelPersistence
0123 1.6 921 19
TestSkipOverseerOperations.testSkipLeaderOperations
0123 0.6 905 11 TestStressLiveNodes.testStress
0123 1.1 651 9 TestTlogReplica.testKillTlogReplica
0123 4.2 112 11
TestXYMultiPolygonShapeQueries.testRandomBig
************ Will BadApple all tests above this line except ones listed at the
top**************
DO NOT ENABLE LIST:
MoveReplicaHDFSTest.testFailedMove
MoveReplicaHDFSTest.testNormalFailedMove
TestControlledRealTimeReopenThread.testCRTReopen
TestICUNormalizer2CharFilter.testRandomStrings
TestICUTokenizerCJK
TestImpersonationWithHadoopAuth.testForwarding
TestLTRReRankingPipeline.testDifferentTopN
TestRandomChains
DO NOT ANNOTATE LIST
CdcrBidirectionalTest.testBiDir
IndexSizeTriggerTest.testMergeIntegration
IndexSizeTriggerTest.testMixedBounds
IndexSizeTriggerTest.testSplitIntegration
IndexSizeTriggerTest.testTrigger
InfixSuggestersTest.testShutdownDuringBuild
ShardSplitTest.test
ShardSplitTest.testSplitMixedReplicaTypes
ShardSplitTest.testSplitWithChaosMonkey
TestLatLonShapeQueries.testRandomBig
TestRandomChains.testRandomChainsWithLargeStrings
TestTriggerIntegration.testSearchRate
Processing file (History bit 3): HOSS-2019-11-18.csv
Processing file (History bit 2): HOSS-2019-11-11.csv
Processing file (History bit 1): HOSS-2019-11-04.csv
Processing file (History bit 0): HOSS-2019-10-28.csv
Number of AwaitsFix: 38 Number of BadApples: 10
Raw fail count by week totals, most recent week first (corresponds to bits):
Week: 0 had 56 failures
Week: 1 had 66 failures
Week: 2 had 83 failures
Week: 3 had 56 failures
**Annotated tests that didn't fail in the last 4 weeks.
**Tests removed from the next two lists because they were specified in
'doNotEnable' in the properties file
no tests removed
**Annotations will be removed from the following tests because they haven't
failed in the last 4 rollups.
**Methods: 3
FullSolrCloudDistribCmdsTest.test
SolrZkClientTest.testSimpleUpdateACLs
TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeader
********Failures in Hoss' reports for the last 4 rollups.
There were 154 unannotated tests that failed in Hoss' rollups. Ordered by the
date I downloaded the rollup file, newest->oldest. See above for the dates the
files were collected
These tests were NOT BadApple'd or AwaitsFix'd
All tests that failed 4 weeks running will be BadApple'd unless there are
objections
Failures in the last 4 reports..
Report Pct runs fails test
0123 3.3 917 45 BasicAuthIntegrationTest.testBasicAuth
0123 0.5 943 7
DimensionalRoutedAliasUpdateProcessorTest.testCatTime
0123 2.7 943 18
DimensionalRoutedAliasUpdateProcessorTest.testTimeCat
0123 8.3 961 91
LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud
0123 43.8 202 85 MoveReplicaHDFSTest.test
0123 0.5 903 5
ReindexCollectionTest.testSameTargetReindexing
0123 4.8 104 5 ShardSplitTest.testSplitWithChaosMonkey
0123 0.5 918 9 SystemCollectionCompatTest.testBackCompat
0123 4.8 941 52
TestCloudSearcherWarming.testRepFactor1LeaderStartup
0123 2.7 946 77
TestModelManagerPersistence.testFilePersistence
0123 2.2 940 70
TestModelManagerPersistence.testWrapperModelPersistence
0123 1.6 921 19
TestSkipOverseerOperations.testSkipLeaderOperations
0123 0.6 905 11 TestStressLiveNodes.testStress
0123 1.1 651 9 TestTlogReplica.testKillTlogReplica
0123 4.2 112 11
TestXYMultiPolygonShapeQueries.testRandomBig
************ Will BadApple all tests above this line except ones listed at the
top**************
012 5.3 70 3 HdfsBasicDistributedZkTest.test
012 9.4 99 6
TimeRoutedAliasUpdateProcessorTest.testDateMathInStart
01 0.6 385 2
ChaosMonkeySafeLeaderWithPullReplicasTest.test
01 5.3 45 5
HdfsTlogReplayBufferedWhileIndexingTest.test
01 1.6 394 4 SplitShardTest.testLiveSplit
01 1.1 391 3
TestConcurrentMergeScheduler.testFlushExceptions
01 0.5 390 2
TestPullReplicaErrorHandling.testPullReplicaDisconnectsFromZooKeeper
0 2 1.1 501 5 HttpPartitionWithTlogReplicasTest.test
0 2 0.6 435 3 TestSimScenario.testSuggestions
0 3 0.5 385 2 MathExpressionTest.testZipFDistribution
0 3 0.6 373 6
SpellCheckCollatorTest.testEstimatedHitCounts
0 3 0.5 397 26 TestDistributedGrouping.test
0 0.5 184 1 AnalyticsMergeStrategyTest.test
0 0.6 181 1 BasicAuthOnSingleNodeTest.basicTest
0 0.6 180 1
ChaosMonkeyNothingIsSafeWithPullReplicasTest.test
0 0.6 179 1
CollectionsAPISolrJTest.testCreateCollWithDefaultClusterPropertiesNewFormat
0 0.5 185 1 DistributedIntervalFacetingTest.test
0 0.5 182 1 ExitableDirectoryReaderTest.testPrefixQuery
0 0.6 181 1 GraphExpressionTest.testScoreNodesStream
0 57.6 132 76 PackageManagerCLITest.testPackageManager
0 4.8 21 1 RestartWhileUpdatingTest.test
0 0.6 178 1
ShardSplitTest.testSplitMixedReplicaTypesLink
0 1.1 182 2
TestBlockJoin.testMultiChildQueriesOfDiffParentLevels
0 0.5 183 1 TestDistributedMissingSort.test
0 18.2 121 22 TestFstDirectAddressing.testDeDupTails
0 0.8 131 1 TestHdfsCloudBackupRestore.test
0 0.6 181 1 TestIndexWriterDelete.testUpdatesOnDiskFull
0 0.6 181 1
TestLucene80DocValuesFormat.testSortedNumericsSingleValuedVsStoredFields
0 3.3 120 4 TestReplicationHandler.testEmptyBackups
0 0.5 183 1
TestSimpleTextTermVectorsFormat.testRamBytesUsed
0 4.0 125 5 TestSolrCachePerf.testGetPutCompute
0 0.6 174 1 TestTlogReplica.testKillLeader
0 0.6 174 1 TestTlogReplica.testRealTimeGet
123 3.2 732 12 RollingRestartTest.test
123 1.0 742 13 SyncSliceTest.test
123 0.5 725 5
TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast
123 1.0 714 4
TestSolrCloudWithDelegationTokens.testDelegationTokenRenew
12 1.1 468 3
TestDelegationWithHadoopAuth.testDelegationTokenRenew
12 2.9 513 8 TestInPlaceUpdatesDistrib.test
12 0.5 518 3 TestPackedInts.testPackedLongValues
12 0.5 471 3 TestTlogReplica.testOnlyLeaderIndexes
12 37.6 453 201
ZkShardTermsTest.testCoreRemovalWhileRecovering
1 3 1.4 418 6
MissingSegmentRecoveryTest.testLeaderRecovery
1 3 23.3 51 8 StressHdfsTest.test
1 3 0.5 296 3
TestTlogReplica.testOutOfOrderDBQWithInPlaceUpdates
1 3 0.5 399 2 TestV2Request.testHttpSolrClient
1 3.8 26 1
CdcrReplicationHandlerTest.testReplicationWithBufferedUpdates
1 1.0 206 2
ConcurrentUpdateHttp2SolrClientTest.testConcurrentUpdate
1 3.4 29 1 HdfsBasicDistributedZk2Test.test
1 3.8 26 1
HdfsCollectionsAPIDistributedZkTest.testCollectionsAPI
1 10.0 30 3 HdfsWriteToMultipleCollectionsTest.test
1 0.5 202 1 LeaderVoteWaitTimeoutTest.basicTest
1 0.5 207 1 OverviewImplTest.testIsOptimized
1 1.0 105 1
TestDirectDocValuesFormat.testSortedNumericsMultipleValuesVsStoredFields
1 0.5 204 1
TestDocumentsWriterStallControl.testAccquireReleaseRace
1 0.5 206 1
TestReplicationHandler.doTestIndexFetchWithMasterUrl
1 1.4 207 3
TestSearcherManager.testConcurrentIndexCloseSearchAndRefresh
1 0.5 199 1
TestSimTriggerIntegration.testNodeLostTriggerRestoreState
1 2.4 206 5 TestStressCloudBlindAtomicUpdates.test_dv
1 2.4 206 5
TestStressCloudBlindAtomicUpdates.test_dv_idx
1 2.4 206 5
TestStressCloudBlindAtomicUpdates.test_dv_stored
1 2.4 206 5
TestStressCloudBlindAtomicUpdates.test_dv_stored_idx
1 2.4 206 5
TestStressCloudBlindAtomicUpdates.test_stored_idx
1 1.0 203 2 TestTlogReplica.testDeleteById
1 3.4 29 1 TestXYPolygonShapeQueries.testRandomBig
23 0.3 505 2 AutoScalingHandlerTest.testReadApi
23 3.6 51 2 HdfsChaosMonkeyNothingIsSafeTest.test
23 0.6 519 3
HealthCheckHandlerTest.testHealthCheckHandler
23 0.6 523 3 MathExpressionTest.testGammaDistribution
23 3.4 52 2 Test2BPostings.test
23 0.4 412 3
TestDistribPackageStore.testPackageStoreManagement
23 0.3 515 2 TestLBHttpSolrClient.testReliability
23 0.3 504 2
TestPullReplicaErrorHandling.testCantConnectToLeader
2 0.3 312 1 AssignBackwardCompatibilityTest.test
2 0.3 311 1
CdcrVersionReplicationTest.testCdcrDocVersions
2 1.0 314 3
CollectionsAPISolrJTest.testCreateCollWithDefaultClusterPropertiesOldFormat
2 0.3 321 1 HealthCheckHandlerTest.testHealthCheckV2Api
2 0.3 316 1 HttpPartitionTest.test
2 1.6 322 5 LeaderFailoverAfterPartitionTest.test
2 0.3 311 1 PeerSyncReplicationTest.test
2 0.3 317 1 ShardSplitTest.testSplitShardWithRule
2 0.3 317 1 ShardSplitTest.testSplitShardWithRuleLink
2 22.3 130 29
SolrPluginUtilsTest.testMinShouldMatchBadQueries
2 2.2 323 7
TestBadConfig.testSchemaMutableButNotManaged
2 0.3 314 1 TestIndexFileDeleter.testExcInDecRef
2 0.3 313 1 TestIndexWriterDelete.testDeletesOnDiskFull
2 0.3 314 1
TestIndexingSequenceNumbers.testStressConcurrentAddAndDeleteAndCommit
2 0.3 313 1 TestLFUCache.testTimeDecayParams
2 0.3 312 1
TestLucene80DocValuesFormat.testSortedNumericBlocksOfVariousBitsPerValue
2 0.3 313 1
TestLucene80DocValuesFormat.testTermsEnumLongSharedPrefixes
2 0.3 312 1
TestLucene80DocValuesFormat.testTermsEnumVariableWidth
2 0.3 315 1 TestOfflineSorter.testThreadSafety
2 0.3 314 1
TestPerFieldDocValuesFormat.testSparseIntNumericsVsStoredFields
2 0.3 315 1 TestPrepRecovery.testLeaderNotResponding
2 0.3 315 1 TestPrepRecovery.testLeaderUnloaded
2 0.3 313 1
TestRandomChains.testRandomChainsWithLargeStrings
2 0.3 310 1 TestRecovery.stressLogReplay
2 0.3 309 1
TestSimTriggerIntegration.testTriggerThrottling
2 0.7 268 2 TestTlogReplica.testRecovery
2 2.4 328 8
UpdateRequestProcessorFactoryTest.testConfiguration
3 1.0 203 2 ChaosMonkeyNothingIsSafeTest.test
3 0.5 197 1
CollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection
3 2.0 199 4 DistributedTermsComponentTest.test
3 5.3 19 1 HdfsSyncSliceTest.test
3 4.2 24 1 HdfsUnloadDistributedZkTest.test
3 1.4 209 3 OverseerRolesTest.testOverseerRole
3 0.5 198 1 PeerSyncTest.test
3 0.5 200 1
TestIndexWriterWithThreads.testIOExceptionDuringAbortWithThreadsOnlyOnce
3 0.5 196 1
TestLucene70NormsFormat.testFewSparseLargeValues
3 0.5 197 1
TestPullReplicaErrorHandling.testCantConnectToPullReplica
3 0.5 198 1
TestReplicationHandler.doTestStressReplication
3 6.5 93 6 TestTlogReplica.testAddDocs
3 2.2 93 2 TestTlogReplica.testRemoveLeader
3 0.5 196 1 UnloadDistributedZkTest.test
***********************Annotations report
Number of AwaitsFix: 38 Number of BadApples: 10
*****************AwaitsFix Annotations:
************Lucene AwaitsFix
TestControlledRealTimeReopenThread.testCRTReopen
@AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/LUCENE-5737";)
TestICUNormalizer2CharFilter.testRandomStrings
@AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/LUCENE-5595";)
TestICUTokenizerCJK.TestICUTokenizerCJK suite
@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/LUCENE-8222";)
TestMoreLikeThis.testMultiFieldShouldReturnPerFieldBooleanQuery
@AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/LUCENE-7161";)
************Solr AwaitsFix
AutoAddReplicasIntegrationTest.testRapidStopStartStopWithPropChange
@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-13811";)
CdcrBidirectionalTest.testBiDir
@AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/SOLR-12524";)
CdcrBootstrapTest.testBootstrapWithContinousIndexingOnSourceCluster
@AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/SOLR-12028";)
CollectionsAPISolrJTest.testCreateAndDeleteShard
@AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/SOLR-13021";)
ComputePlanActionTest.testNodeLost
@LuceneTestCase.AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";)
ConcurrentCreateRoutedAliasTest.ConcurrentCreateRoutedAliasTest suite
@LuceneTestCase.AwaitsFix(bugUrl =
"https://issues.apache.org/jira/browse/SOLR-12386";) // "Can't find resource"
LeaderTragicEventTest.LeaderTragicEventTest suite
@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-13237";)
MoveReplicaHDFSTest.testFailedMove
@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12080";) //
added 03-Oct-2018
ReplaceNodeNoTargetTest.ReplaceNodeNoTargetTest suite
@LuceneTestCase.AwaitsFix(bugUrl =
"https://issues.apache.org/jira/browse/SOLR-11067";)
ReplaceNodeNoTargetTest.test
@LuceneTestCase.AwaitsFix(bugUrl =
"https://issues.apache.org/jira/browse/SOLR-11067";)
SSLMigrationTest.SSLMigrationTest suite
@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";) //
17-Mar-2018
SaslZkACLProviderTest.testSaslZkACLProvider
@AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/SOLR-13075";)
ScheduledMaintenanceTriggerTest.testInactiveShardCleanup
@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";) //
17-Mar-2018
ScheduledTriggerTest.testTrigger
@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";) //
added 20-Sep-2018
SearchRateTriggerIntegrationTest.testDeleteNode
@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-13163";)
SearchRateTriggerTest.testTrigger
@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";)
SolrExporterIntegrationTest.SolrExporterIntegrationTest suite
@org.apache.lucene.util.LuceneTestCase.AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-13786";)
Tagger2Test.testStopWords
@AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/LUCENE-8344";)
TestCollapseQParserPlugin.testStringCollapse
@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-11974";)
TestHierarchicalDocBuilder.testThreeLevelHierarchy
@AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/SOLR-12801";) //
this test fails easily under beasting
TestImpersonationWithHadoopAuth.testForwarding
@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/HADOOP-9893";)
TestJsonFacetRefinement.testProcessEmptyRefinement
@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12556";)
TestJsonFacetRefinement.testSortedSubFacetRefinementWhenParentOnlyReturnedByOneShardProcessEmpty
@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12556";)
TestLTRReRankingPipeline.testDifferentTopN
@AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/SOLR-11134";)
TestMinMaxOnMultiValuedField.testDoubleFieldCache
@AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/LUCENE-6709";)
TestMinMaxOnMultiValuedField.testFloatFieldCache
@AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/LUCENE-6709";)
TestMinMaxOnMultiValuedField.testIntFieldCache
@AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/LUCENE-6709";)
TestMinMaxOnMultiValuedField.testLongFieldCache
@AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/LUCENE-6709";)
TestPullReplica.TestPullReplica suite
@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";)
TestSimComputePlanAction.testNodeLost
@AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/SOLR-12028";) //
if you beast this, eventually you will see
TestSimExtremeIndexing.TestSimExtremeIndexing suite
@org.apache.lucene.util.LuceneTestCase.AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12923";)
TestSimLargeCluster.testBasic
@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";) //
this test hits a timeout easily
TestStressRecovery.TestStressRecovery suite
@LuceneTestCase.AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";)
// 6-Sep-2018
TriLevelCompositeIdRoutingTest.test
@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-13369";)
*****************BadApple Annotations:
************Lucene BadApple
************Solr BadApple
FullSolrCloudDistribCmdsTest.test
@BadApple(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";) //
annotated on: 24-Dec-2018
HdfsNNFailoverTest.HdfsNNFailoverTest suite
@LuceneTestCase.BadApple(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";)
// 12-Jun-2018
HdfsRecoveryZkTest.HdfsRecoveryZkTest suite
@BadApple(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";)
MoveReplicaHDFSTest.testNormalFailedMove
@BadApple(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";) //
14-Oct-2018
MultiThreadedOCPTest.test
@BadApple(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";) //
annotated on: 24-Dec-2018
SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
@LuceneTestCase.BadApple(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";)
// added 20-Jul-2018
SolrZkClientTest.testSimpleUpdateACLs
@BadApple(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";) //
annotated on: 24-Dec-2018
TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeader
@BadApple(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";) //
6-Sep-2018
TimeRoutedAliasUpdateProcessorTest.TimeRoutedAliasUpdateProcessorTest suite
@LuceneTestCase.BadApple(bugUrl =
"https://issues.apache.org/jira/browse/SOLR-13059";)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
