Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/7055/
Java: 64bit/jdk1.8.0_144 -XX:-UseCompressedOops -XX:+UseParallelGC

6 tests failed.
FAILED:  
junit.framework.TestSuite.org.apache.solr.handler.dataimport.TestTemplateTransformer

Error Message:
Could not remove the following files (in the order of attempts):    
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\contrib\solr-dataimporthandler\test\J1\temp\solr.handler.dataimport.TestTemplateTransformer_7829E85B5E0366EC-001\dih-properties-001:
 java.nio.file.AccessDeniedException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\contrib\solr-dataimporthandler\test\J1\temp\solr.handler.dataimport.TestTemplateTransformer_7829E85B5E0366EC-001\dih-properties-001
 

Stack Trace:
java.io.IOException: Could not remove the following files (in the order of 
attempts):
   
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\contrib\solr-dataimporthandler\test\J1\temp\solr.handler.dataimport.TestTemplateTransformer_7829E85B5E0366EC-001\dih-properties-001:
 java.nio.file.AccessDeniedException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\contrib\solr-dataimporthandler\test\J1\temp\solr.handler.dataimport.TestTemplateTransformer_7829E85B5E0366EC-001\dih-properties-001

        at __randomizedtesting.SeedInfo.seed([7829E85B5E0366EC]:0)
        at org.apache.lucene.util.IOUtils.rm(IOUtils.java:329)
        at 
org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:216)
        at 
com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestCloudRecovery

Error Message:
Could not remove the following files (in the order of attempts):    
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard2_replica_n6\data\tlog\tlog.0000000000000000001:
 java.nio.file.FileSystemException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard2_replica_n6\data\tlog\tlog.0000000000000000001:
 The process cannot access the file because it is being used by another 
process.     
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard2_replica_n6\data\tlog:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard2_replica_n6\data\tlog
    
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard2_replica_n6\data:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard2_replica_n6\data
    
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard2_replica_n6:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard2_replica_n6
    
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard1_replica_n2\data\tlog\tlog.0000000000000000001:
 java.nio.file.FileSystemException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard1_replica_n2\data\tlog\tlog.0000000000000000001:
 The process cannot access the file because it is being used by another 
process.     
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard1_replica_n2\data\tlog:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard1_replica_n2\data\tlog
    
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard1_replica_n2\data:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard1_replica_n2\data
    
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard1_replica_n2:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard1_replica_n2
    
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1
    
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard2_replica_n4\data\tlog\tlog.0000000000000000001:
 java.nio.file.FileSystemException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard2_replica_n4\data\tlog\tlog.0000000000000000001:
 The process cannot access the file because it is being used by another 
process.     
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard2_replica_n4\data\tlog:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard2_replica_n4\data\tlog
    
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard2_replica_n4\data:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard2_replica_n4\data
    
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard2_replica_n4:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard2_replica_n4
    
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard1_replica_n1\data\tlog\tlog.0000000000000000001:
 java.nio.file.FileSystemException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard1_replica_n1\data\tlog\tlog.0000000000000000001:
 The process cannot access the file because it is being used by another 
process.     
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard1_replica_n1\data\tlog:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard1_replica_n1\data\tlog
    
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard1_replica_n1\data:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard1_replica_n1\data
    
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard1_replica_n1:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard1_replica_n1
    
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2
    
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001
    
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001
 

Stack Trace:
java.io.IOException: Could not remove the following files (in the order of 
attempts):
   
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard2_replica_n6\data\tlog\tlog.0000000000000000001:
 java.nio.file.FileSystemException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard2_replica_n6\data\tlog\tlog.0000000000000000001:
 The process cannot access the file because it is being used by another process.

   
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard2_replica_n6\data\tlog:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard2_replica_n6\data\tlog
   
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard2_replica_n6\data:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard2_replica_n6\data
   
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard2_replica_n6:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard2_replica_n6
   
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard1_replica_n2\data\tlog\tlog.0000000000000000001:
 java.nio.file.FileSystemException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard1_replica_n2\data\tlog\tlog.0000000000000000001:
 The process cannot access the file because it is being used by another process.

   
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard1_replica_n2\data\tlog:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard1_replica_n2\data\tlog
   
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard1_replica_n2\data:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard1_replica_n2\data
   
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard1_replica_n2:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1\collection1_shard1_replica_n2
   
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node1
   
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard2_replica_n4\data\tlog\tlog.0000000000000000001:
 java.nio.file.FileSystemException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard2_replica_n4\data\tlog\tlog.0000000000000000001:
 The process cannot access the file because it is being used by another process.

   
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard2_replica_n4\data\tlog:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard2_replica_n4\data\tlog
   
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard2_replica_n4\data:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard2_replica_n4\data
   
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard2_replica_n4:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard2_replica_n4
   
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard1_replica_n1\data\tlog\tlog.0000000000000000001:
 java.nio.file.FileSystemException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard1_replica_n1\data\tlog\tlog.0000000000000000001:
 The process cannot access the file because it is being used by another process.

   
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard1_replica_n1\data\tlog:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard1_replica_n1\data\tlog
   
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard1_replica_n1\data:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard1_replica_n1\data
   
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard1_replica_n1:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2\collection1_shard1_replica_n1
   
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001\node2
   
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001\tempDir-001
   
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestCloudRecovery_9E68C9EAEE2F34B1-001

        at __randomizedtesting.SeedInfo.seed([9E68C9EAEE2F34B1]:0)
        at org.apache.lucene.util.IOUtils.rm(IOUtils.java:329)
        at 
org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:216)
        at 
com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testEventQueue

Error Message:
action wasn't interrupted

Stack Trace:
java.lang.AssertionError: action wasn't interrupted
        at 
__randomizedtesting.SeedInfo.seed([9E68C9EAEE2F34B1:57DD8B44E748F244]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at 
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testEventQueue(TestTriggerIntegration.java:638)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.core.TestLazyCores.testNoCommit

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
        at 
__randomizedtesting.SeedInfo.seed([9E68C9EAEE2F34B1:4108683B25085714]:0)
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:901)
        at org.apache.solr.core.TestLazyCores.check10(TestLazyCores.java:855)
        at 
org.apache.solr.core.TestLazyCores.testNoCommit(TestLazyCores.java:832)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: REQUEST FAILED: 
xpath=//result[@numFound='10']
        xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int 
name="QTime">0</int><lst name="params"><str 
name="q">*:*</str></lst></lst><result name="response" numFound="0" 
start="0"></result>
</response>

        request was:q=*:*
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:894)
        ... 41 more


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestLazyCores

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.core.TestLazyCores:     1) 
Thread[id=10489, name=searcherExecutor-4594-thread-1, state=WAITING, 
group=TGRP-TestLazyCores]         at sun.misc.Unsafe.park(Native Method)        
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE 
scope at org.apache.solr.core.TestLazyCores: 
   1) Thread[id=10489, name=searcherExecutor-4594-thread-1, state=WAITING, 
group=TGRP-TestLazyCores]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([9E68C9EAEE2F34B1]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestLazyCores

Error Message:
There are still zombie threads that couldn't be terminated:    1) 
Thread[id=10489, name=searcherExecutor-4594-thread-1, state=WAITING, 
group=TGRP-TestLazyCores]         at sun.misc.Unsafe.park(Native Method)        
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   1) Thread[id=10489, name=searcherExecutor-4594-thread-1, state=WAITING, 
group=TGRP-TestLazyCores]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([9E68C9EAEE2F34B1]:0)




Build Log:
[...truncated 11440 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration
   [junit4]   2> Creating dataDir: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.autoscaling.sim.TestTriggerIntegration_9E68C9EAEE2F34B1-001\init-core-data-001
   [junit4]   2> 9965 WARN  
(SUITE-TestTriggerIntegration-seed#[9E68C9EAEE2F34B1]-worker) [    ] 
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 9966 INFO  
(SUITE-TestTriggerIntegration-seed#[9E68C9EAEE2F34B1]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 9978 INFO  
(SUITE-TestTriggerIntegration-seed#[9E68C9EAEE2F34B1]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: 
@org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 9978 INFO  
(SUITE-TestTriggerIntegration-seed#[9E68C9EAEE2F34B1]-worker) [    ] 
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 10018 INFO  
(SUITE-TestTriggerIntegration-seed#[9E68C9EAEE2F34B1]-worker) [    ] 
o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find 
system property or JNDI)
   [junit4]   2> 10065 INFO  
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9E68C9EAEE2F34B1]) [    
] o.a.s.SolrTestCaseJ4 ###Starting testNodeLostTrigger
   [junit4]   2> 10070 INFO  
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9E68C9EAEE2F34B1]) [    
] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing 
object cache...
   [junit4]   2> 10070 DEBUG 
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9E68C9EAEE2F34B1]) [    
] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed 
explicitly
   [junit4]   2> 10110 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 10110 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 10121 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 2
   [junit4]   2> 10131 DEBUG (simCloudManagerPool-15-thread-1) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 10273 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10000_solr]
   [junit4]   2> 10273 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion -1
   [junit4]   2> 10274 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 10274 DEBUG (simCloudManagerPool-15-thread-1) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
3
   [junit4]   2> 10274 DEBUG (simCloudManagerPool-15-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10000_solr]
   [junit4]   2> 10274 DEBUG (simCloudManagerPool-15-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10000_solr]
   [junit4]   2> 10292 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 10293 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 2
   [junit4]   2> 10293 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 10295 INFO  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 10295 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 10295 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 10299 DEBUG (ScheduledTrigger-18-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 10300 DEBUG (ScheduledTrigger-18-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 1
   [junit4]   2> 10300 DEBUG (ScheduledTrigger-18-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10000_solr
   [junit4]   2> 10300 DEBUG (ScheduledTrigger-18-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10000_solr
   [junit4]   2> 10322 DEBUG (ScheduledTrigger-18-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 1
   [junit4]   2> 10322 DEBUG (ScheduledTrigger-18-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 10322 DEBUG (ScheduledTrigger-18-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost 
nodes: [127.0.0.1:10000_solr]
   [junit4]   2> 10323 DEBUG (ScheduledTrigger-18-thread-2) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"3bc8ce36c7236Tnaka5ov337l12fifkg2hnn9p",
   [junit4]   2>   "source":"node_lost_trigger",
   [junit4]   2>   "eventTime":1051738227110454,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1051738227110454],
   [junit4]   2>     "nodeNames":["127.0.0.1:10000_solr"]}}
   [junit4]   2> 10330 DEBUG (AutoscalingActionExecutor-19-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"3bc8ce36c7236Tnaka5ov337l12fifkg2hnn9p",
   [junit4]   2>   "source":"node_lost_trigger",
   [junit4]   2>   "eventTime":1051738227110454,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1051738227110454],
   [junit4]   2>     "_enqueue_time_":1051739339174754,
   [junit4]   2>     "nodeNames":["127.0.0.1:10000_solr"]}}
   [junit4]   2> 10335 DEBUG (AutoscalingActionExecutor-19-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 240 ms for event 
id=3bc8ce36c7236Tnaka5ov337l12fifkg2hnn9p
   [junit4]   2> 10335 DEBUG (simCloudManagerPool-15-thread-3) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 10337 DEBUG (simCloudManagerPool-15-thread-3) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
4
   [junit4]   2> 10337 DEBUG (simCloudManagerPool-15-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr]
   [junit4]   2> 10337 DEBUG (simCloudManagerPool-15-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr]
   [junit4]   2> 10337 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 10338 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 10338 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 10342 DEBUG (ScheduledTrigger-18-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 10351 DEBUG (ScheduledTrigger-18-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 1
   [junit4]   2> 10363 DEBUG (ScheduledTrigger-18-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 10372 DEBUG (ScheduledTrigger-18-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 1
   [junit4]   2> 10379 INFO  
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9E68C9EAEE2F34B1]) [    
] o.a.s.SolrTestCaseJ4 ###Ending testNodeLostTrigger
   [junit4]   2> 10382 INFO  
(TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[9E68C9EAEE2F34B1])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting testEventFromRestoredState
   [junit4]   2> 10382 DEBUG 
(TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[9E68C9EAEE2F34B1])
 [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 5
   [junit4]   2> 10383 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
   [junit4]   2> 10383 INFO  
(TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[9E68C9EAEE2F34B1])
 [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and 
clearing object cache...
   [junit4]   2> 10383 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 10383 DEBUG 
(TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[9E68C9EAEE2F34B1])
 [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed 
explicitly
   [junit4]   2> 10383 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 10387 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 10387 WARN  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2>        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
   [junit4]   2>        at 
java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:154)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 10387 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 6
   [junit4]   2> 10393 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10002_solr]
   [junit4]   2> 10393 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion -1
   [junit4]   2> 10393 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
   [junit4]   2> 10394 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 10394 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
   [junit4]   2> 10394 DEBUG (simCloudManagerPool-15-thread-5) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 10396 DEBUG (simCloudManagerPool-15-thread-5) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
7
   [junit4]   2> 10396 DEBUG (simCloudManagerPool-15-thread-5) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10002_solr]
   [junit4]   2> 10397 DEBUG (ScheduledTrigger-20-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 10400 DEBUG (simCloudManagerPool-15-thread-5) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10002_solr]
   [junit4]   2> 10400 DEBUG (simCloudManagerPool-15-thread-5) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated 
with properties: {event=nodeAdded, waitFor=10, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}],
 enabled=true}
   [junit4]   2> 10400 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
   [junit4]   2> 10401 INFO  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 10401 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
   [junit4]   2> 10402 DEBUG (ScheduledTrigger-20-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 10403 DEBUG (ScheduledTrigger-20-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 10403 DEBUG (ScheduledTrigger-20-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10003_solr at time 
1051743411837254
   [junit4]   2> 10419 DEBUG (ScheduledTrigger-20-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 10425 DEBUG (ScheduledTrigger-20-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 10425 DEBUG (ScheduledTrigger-20-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 10440 DEBUG (ScheduledTrigger-20-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 10447 DEBUG (ScheduledTrigger-20-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 10447 DEBUG (ScheduledTrigger-20-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 10461 DEBUG (ScheduledTrigger-20-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 10467 DEBUG (ScheduledTrigger-20-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 10467 DEBUG (ScheduledTrigger-20-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 10482 DEBUG (ScheduledTrigger-20-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 10489 DEBUG (ScheduledTrigger-20-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 10489 DEBUG (ScheduledTrigger-20-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 10503 DEBUG (ScheduledTrigger-20-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 10510 DEBUG (ScheduledTrigger-20-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 10510 DEBUG (ScheduledTrigger-20-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 10523 DEBUG (ScheduledTrigger-20-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 10531 DEBUG (ScheduledTrigger-20-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 10531 DEBUG (ScheduledTrigger-20-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 10544 DEBUG (ScheduledTrigger-20-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 10551 DEBUG (ScheduledTrigger-20-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 10551 DEBUG (ScheduledTrigger-20-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 10567 DEBUG (ScheduledTrigger-20-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 10575 DEBUG (ScheduledTrigger-20-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 10575 DEBUG (ScheduledTrigger-20-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 10588 DEBUG (ScheduledTrigger-20-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 10596 DEBUG (ScheduledTrigger-20-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 10596 DEBUG (ScheduledTrigger-20-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 10610 DEBUG (ScheduledTrigger-20-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 10618 DEBUG (ScheduledTrigger-20-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 10618 DEBUG (ScheduledTrigger-20-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 10618 DEBUG (ScheduledTrigger-20-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing 
registered processor for nodes: [127.0.0.1:10003_solr] added at times 
[1051743411837254], now=1051754132494554
   [junit4]   2> 10618 DEBUG (ScheduledTrigger-20-thread-3) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"3bc8e18751946Tnaka5ov337l12fifkg2hnn9s",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":1051743411837254,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1051743411837254],
   [junit4]   2>     "nodeNames":["127.0.0.1:10003_solr"]}}
   [junit4]   2> 10621 DEBUG (AutoscalingActionExecutor-21-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"3bc8e18751946Tnaka5ov337l12fifkg2hnn9s",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":1051743411837254,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1051743411837254],
   [junit4]   2>     "_enqueue_time_":1051754152192304,
   [junit4]   2>     "nodeNames":["127.0.0.1:10003_solr"]}}
   [junit4]   2> 10622 DEBUG (AutoscalingActionExecutor-21-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 72 ms for event 
id=3bc8e18751946Tnaka5ov337l12fifkg2hnn9s
   [junit4]   2> 10630 DEBUG (ScheduledTrigger-20-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 10641 DEBUG (ScheduledTrigger-20-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 10641 DEBUG (ScheduledTrigger-20-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 10641 DEBUG (ScheduledTrigger-20-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10004_solr at time 
1051755283797954
   [junit4]   2> 10651 DEBUG (ScheduledTrigger-20-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 10662 DEBUG (ScheduledTrigger-20-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 10662 DEBUG (ScheduledTrigger-20-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 10673 DEBUG (ScheduledTrigger-20-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 10683 DEBUG (ScheduledTrigger-20-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 10683 DEBUG (ScheduledTrigger-20-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 10694 DEBUG (ScheduledTrigger-20-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 10713 DEBUG (ScheduledTrigger-20-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 10713 DEBUG (ScheduledTrigger-20-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 10715 DEBUG (ScheduledTrigger-20-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 10735 DEBUG (ScheduledTrigger-20-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 10735 DEBUG (ScheduledTrigger-20-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 10735 DEBUG (ScheduledTrigger-20-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 10755 DEBUG (ScheduledTrigger-20-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 10755 DEBUG (ScheduledTrigger-20-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 10755 DEBUG (ScheduledTrigger-20-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 10777 DEBUG (ScheduledTrigger-20-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 10777 DEBUG (ScheduledTrigger-20-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 10777 DEBUG (ScheduledTrigger-20-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 10798 DEBUG (ScheduledTrigger-20-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 10798 DEBUG (ScheduledTrigger-20-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 10798 DEBUG (ScheduledTrigger-20-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 10819 DEBUG (ScheduledTrigger-20-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 10819 DEBUG (ScheduledTrigger-20-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 10819 DEBUG (ScheduledTrigger-20-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 10822 INFO  
(TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[9E68C9EAEE2F34B1])
 [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and 
clearing object cache...
   [junit4]   2> 10822 WARN  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2>        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
   [junit4]   2>        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:166)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 10822 DEBUG 
(TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[9E68C9EAEE2F34B1])
 [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed 
explicitly
   [junit4]   2> 10824 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 7
   [junit4]   2> 10824 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10004_solr, 127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
   [junit4]   2> 10824 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10004_solr, 127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
   [junit4]   2> 10824 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated 
with properties: {event=nodeAdded, waitFor=10, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}],
 enabled=true}
   [junit4]   2> 10824 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion -1
   [junit4]   2> 10824 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
   [junit4]   2> 10824 INFO  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 10825 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeAddedTrigger Adding node from marker path: 127.0.0.1:10004_solr
   [junit4]   2> 10825 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
   [junit4]   2> 10825 DEBUG (ScheduledTrigger-22-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 10825 DEBUG (ScheduledTrigger-22-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 10825 DEBUG (ScheduledTrigger-22-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 10846 DEBUG (ScheduledTrigger-22-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 10846 DEBUG (ScheduledTrigger-22-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 10847 DEBUG (ScheduledTrigger-22-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing 
registered processor for nodes: [127.0.0.1:10004_solr] added at times 
[1051755283797954], now=1051765561831104
   [junit4]   2> 10846 DEBUG (ScheduledTrigger-22-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 10847 DEBUG (ScheduledTrigger-22-thread-3) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"3bc90dc14d7c2Tnaka5ov337l12fifkg2hnn9v",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":1051755283797954,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1051755283797954],
   [junit4]   2>     "nodeNames":["127.0.0.1:10004_solr"]}}
   [junit4]   2> 10848 DEBUG (AutoscalingActionExecutor-23-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"3bc90dc14d7c2Tnaka5ov337l12fifkg2hnn9v",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":1051755283797954,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1051755283797954],
   [junit4]   2>     "_enqueue_time_":1051765583648004,
   [junit4]   2>     "nodeNames":["127.0.0.1:10004_solr"]}}
   [junit4]   2> 10848 INFO  
(TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[9E68C9EAEE2F34B1])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testEventFromRestoredState
   [junit4]   2> 10851 DEBUG (AutoscalingActionExecutor-23-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 149 ms for event 
id=3bc90dc14d7c2Tnaka5ov337l12fifkg2hnn9v
   [junit4]   2> 10855 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[9E68C9EAEE2F34B1])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeAddedTriggerRestoreState
   [junit4]   2> 10855 DEBUG 
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[9E68C9EAEE2F34B1])
 [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 8
   [junit4]   2> 10859 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[9E68C9EAEE2F34B1])
 [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and 
clearing object cache...
   [junit4]   2> 10860 DEBUG 
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[9E68C9EAEE2F34B1])
 [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed 
explicitly
   [junit4]   2> 10860 WARN  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are 
closed, exiting.
   [junit4]   2> 10862 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 10863 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 9
   [junit4]   2> 10863 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10004_solr, 127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
   [junit4]   2> 10863 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion -1
   [junit4]   2> 10863 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
   [junit4]   2> 10863 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 10863 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
   [junit4]   2> 10867 DEBUG (simCloudManagerPool-15-thread-10) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 10868 DEBUG (simCloudManagerPool-15-thread-10) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
10
   [junit4]   2> 10869 DEBUG (simCloudManagerPool-15-thread-10) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10004_solr, 127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
   [junit4]   2> 10869 DEBUG (ScheduledTrigger-24-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 10869 DEBUG (simCloudManagerPool-15-thread-10) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger 
instantiated with properties: {event=nodeAdded, waitFor=5, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}],
 enabled=true}
   [junit4]   2> 10869 DEBUG (simCloudManagerPool-15-thread-10) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10004_solr, 127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
   [junit4]   2> 10869 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 10
   [junit4]   2> 10870 INFO  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 10870 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 10, lastZnodeVersion 10
   [junit4]   2> 10871 DEBUG (ScheduledTrigger-24-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
   [junit4]   2> 10871 DEBUG (ScheduledTrigger-24-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 10890 DEBUG (ScheduledTrigger-24-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 5
   [junit4]   2> 10893 DEBUG (ScheduledTrigger-24-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
   [junit4]   2> 10893 DEBUG (ScheduledTrigger-24-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 5
   [junit4]   2> 10893 DEBUG (ScheduledTrigger-24-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10005_solr at time 
1051767886198004
   [junit4]   2> 10909 DEBUG (simCloudManagerPool-15-thread-11) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 10909 DEBUG (simCloudManagerPool-15-thread-11) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
11
   [junit4]   2> 10909 DEBUG (simCloudManagerPool-15-thread-11) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10004_solr, 127.0.0.1:10005_solr, 127.0.0.1:10002_solr, 
127.0.0.1:10003_solr]
   [junit4]   2> 10909 DEBUG (simCloudManagerPool-15-thread-11) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger 
instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}],
 enabled=true}
   [junit4]   2> 10909 DEBUG (simCloudManagerPool-15-thread-11) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10004_solr, 127.0.0.1:10005_solr, 127.0.0.1:10002_solr, 
127.0.0.1:10003_solr]
   [junit4]   2> 10909 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 11
   [junit4]   2> 10910 INFO  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 10910 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeAddedTrigger Adding node from marker path: 127.0.0.1:10005_solr
   [junit4]   2> 10910 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 11, lastZnodeVersion 11
   [junit4]   2> 10911 DEBUG (ScheduledTrigger-24-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
   [junit4]   2> 10911 DEBUG (ScheduledTrigger-24-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 5
   [junit4]   2> 10911 DEBUG (ScheduledTrigger-24-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger firing 
registered processor for nodes: [127.0.0.1:10005_solr] added at times 
[1051768726589354], now=1051768783563604
   [junit4]   2> 10911 DEBUG (ScheduledTrigger-24-thread-4) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"3bc93fd558faaTnaka5ov337l12fifkg2hnn9y",
   [junit4]   2>   "source":"node_added_restore_trigger",
   [junit4]   2>   "eventTime":1051768726589354,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1051768726589354],
   [junit4]   2>     "nodeNames":["127.0.0.1:10005_solr"]}}
   [junit4]   2> 10913 DEBUG (AutoscalingActionExecutor-25-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"3bc93fd558faaTnaka5ov337l12fifkg2hnn9y",
   [junit4]   2>   "source":"node_added_restore_trigger",
   [junit4]   2>   "eventTime":1051768726589354,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1051768726589354],
   [junit4]   2>     "_enqueue_time_":1051768796591154,
   [junit4]   2>     "nodeNames":["127.0.0.1:10005_solr"]}}
   [junit4]   2> 10913 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[9E68C9EAEE2F34B1])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeAddedTriggerRestoreState
   [junit4]   2> 10917 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[9E68C9EAEE2F34B1]) [    
] o.a.s.SolrTestCaseJ4 ###Starting testNodeAddedTrigger
   [junit4]   2> 10917 DEBUG 
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[9E68C9EAEE2F34B1]) [    
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 12
   [junit4]   2> 10917 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[9E68C9EAEE2F34B1]) [    
] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing 
object cache...
   [junit4]   2> 10918 WARN  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are 
closed, exiting.
   [junit4]   2> 10918 DEBUG (AutoscalingActionExecutor-25-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 265 ms for event 
id=3bc93fd558faaTnaka5ov337l12fifkg2hnn9y
   [junit4]   2> 10919 DEBUG 
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[9E68C9EAEE2F34B1]) [    
] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed 
explicitly
   [junit4]   2> 10920 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 10921 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 13
   [junit4]   2> 10921 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10004_solr, 127.0.0.1:10005_solr, 127.0.0.1:10002_solr, 
127.0.0.1:10003_solr]
   [junit4]   2> 10921 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 13, lastZnodeVersion -1
   [junit4]   2> 10921 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 13
   [junit4]   2> 10921 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 10921 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 13, lastZnodeVersion 13
   [junit4]   2> 10922 DEBUG (simCloudManagerPool-15-thread-14) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 10922 DEBUG (simCloudManagerPool-15-thread-14) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
14
   [junit4]   2> 10933 DEBUG (ScheduledTrigger-26-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 5
   [junit4]   2> 10935 DEBUG (simCloudManagerPool-15-thread-14) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10004_solr, 127.0.0.1:10005_solr, 127.0.0.1:10002_solr, 
127.0.0.1:10003_solr]
   [junit4]   2> 10935 DEBUG (simCloudManagerPool-15-thread-14) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10004_solr, 127.0.0.1:10005_solr, 127.0.0.1:10002_solr, 
127.0.0.1:10003_solr]
   [junit4]   2> 10935 DEBUG (simCloudManagerPool-15-thread-14) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated 
with properties: {event=nodeAdded, waitFor=3, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}],
 enabled=true}
   [junit4]   2> 10935 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 14
   [junit4]   2> 10936 INFO  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 10936 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 14, lastZnodeVersion 14
   [junit4]   2> 10937 DEBUG (ScheduledTrigger-26-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 10937 DEBUG (ScheduledTrigger-26-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 6
   [junit4]   2> 10937 DEBUG (ScheduledTrigger-26-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10006_solr at time 
1051770077032404
   [junit4]   2> 10955 DEBUG (ScheduledTrigger-26-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 6
   [junit4]   2> 10960 DEBUG (ScheduledTrigger-26-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 10960 DEBUG (ScheduledTrigger-26-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 6
   [junit4]   2> 10977 DEBUG (ScheduledTrigger-26-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 6
   [junit4]   2> 10982 DEBUG (ScheduledTrigger-26-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 10982 DEBUG (ScheduledTrigger-26-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 6
   [junit4]   2> 10998 DEBUG (ScheduledTrigger-26-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 6
   [junit4]   2> 11002 DEBUG (ScheduledTrigger-26-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 11002 DEBUG (ScheduledTrigger-26-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 6
   [junit4]   2> 11003 DEBUG (ScheduledTrigger-26-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing 
registered processor for nodes: [127.0.0.1:10006_solr] added at times 
[1051770077032404], now=1051773367372904
   [junit4]   2> 11003 DEBUG (ScheduledTrigger-26-thread-2) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"3bc944dd3afd4Tnaka5ov337l12fifkg2hnna1",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":1051770077032404,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1051770077032404],
   [junit4]   2>     "nodeNames":["127.0.0.1:10006_solr"]}}
   [junit4]   2> 11004 DEBUG (AutoscalingActionExecutor-27-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"3bc944dd3afd4Tnaka5ov337l12fifkg2hnna1",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":1051770077032404,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1051770077032404],
   [junit4]   2>     "_enqueue_time_":1051773379945354,
   [junit4]   2>     "nodeNames":["127.0.0.1:10006_solr"]}}
   [junit4]   2> 11005 DEBUG (simCloudManagerPool-15-thread-16) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 11006 DEBUG (simCloudManagerPool-15-thread-16) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
15
   [junit4]   2> 11006 DEBUG (simCloudManagerPool-15-thread-16) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10004_solr, 127.0.0.1:10006_solr, 127.0.0.1:10005_solr, 
127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
   [junit4]   2> 11006 DEBUG (simCloudManagerPool-15-thread-16) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10004_solr, 127.0.0.1:10006_solr, 127.0.0.1:10005_solr, 
127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
   [junit4]   2> 11006 DEBUG (simCloudManagerPool-15-thread-16) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated 
with properties: {event=nodeAdded, waitFor=3, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}],
 enabled=true}
   [junit4]   2> 11006 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 15
   [junit4]   2> 11006 DEBUG (AutoscalingActionExecutor-27-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 115 ms for event 
id=3bc944dd3afd4Tnaka5ov337l12fifkg2hnna1
   [junit4]   2> 11006 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 15, lastZnodeVersion 15
   [junit4]   2> 11017 DEBUG (ScheduledTrigger-26-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 6
   [junit4]   2> 11043 DEBUG (ScheduledTrigger-26-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 11043 DEBUG (ScheduledTrigger-26-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 6
   [junit4]   2> 11043 DEBUG (ScheduledTrigger-26-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 6
   [junit4]   2> 11067 DEBUG (ScheduledTrigger-26-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 11067 DEBUG (ScheduledTrigger-26-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 6
   [junit4]   2> 11067 DEBUG (ScheduledTrigger-26-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 6
   [junit4]   2> 11094 DEBUG (ScheduledTrigger-26-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 11094 DEBUG (ScheduledTrigger-26-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 6
   [junit4]   2> 11095 DEBUG (ScheduledTrigger-26-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 6
   [junit4]   2> 11095 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[9E68C9EAEE2F34B1]) [    
] o.a.s.SolrTestCaseJ4 ###Ending testNodeAddedTrigger
   [junit4]   2> 11098 INFO  
(TEST-TestTriggerIntegration.testSearchRate-seed#[9E68C9EAEE2F34B1]) [    ] 
o.a.s.SolrTestCaseJ4 ###Starting testSearchRate
   [junit4]   2> 11098 DEBUG 
(TEST-TestTriggerIntegration.testSearchRate-seed#[9E68C9EAEE2F34B1]) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
16
   [junit4]   2> 11098 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 16
   [junit4]   2> 11118 DEBUG (ScheduledTrigger-26-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 11118 DEBUG (ScheduledTrigger-26-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 6
   [junit4]   2> 11126 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 11126 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 11126 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 16, lastZnodeVersion 16
   [junit4]   2> 11126 INFO  
(TEST-TestTriggerIntegration.testSearchRate-seed#[9E68C9EAEE2F34B1]) [    ] 
o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing 
object cache...
   [junit4]   2> 11126 WARN  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are 
closed, exiting.
   [junit4]   2> 11127 DEBUG 
(TEST-TestTriggerIntegration.testSearchRate-seed#[9E68C9EAEE2F34B1]) [    ] 
o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 11132 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 11133 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 17
   [junit4]   2> 11133 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10004_solr, 127.0.0.1:10006_solr, 127.0.0.1:10005_solr, 
127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
   [junit4]   2> 11133 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 17, lastZnodeVersion -1
   [junit4]   2> 11133 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 17
   [junit4]   2> 11133 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 11134 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 17, lastZnodeVersion 17
   [junit4]   2> 11137 DEBUG (simCloudManagerPool-15-thread-18) [    ] 
o.a.s.c.a.s.SimCloudManager Invoking Collection Action :create with params 
?action=CREATE&name=collection1&collection.configName=conf&numShards=1&replicationFactor=2&nrtReplicas=2
   [junit4]   2> 11139 DEBUG (ScheduledTrigger-28-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 6
   [junit4]   2> 11163 DEBUG (ScheduledTrigger-28-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 6
   [junit4]   2> 11185 DEBUG (ScheduledTrigger-28-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 6
   [junit4]   2> 11206 DEBUG (ScheduledTrigger-28-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 6
   [junit4]   2> 11222 DEBUG (simCloudManagerPool-15-thread-22) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 11222 DEBUG (simCloudManagerPool-15-thread-22) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
18
   [junit4]   2> 11227 DEBUG (simCloudManagerPool-15-thread-22) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10004_solr, 127.0.0.1:10006_solr, 127.0.0.1:10005_solr, 
127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
   [junit4]   2> 11227 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 18
   [junit4]   2> 11227 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 11228 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 18, lastZnodeVersion 18
   [junit4]   2> 11228 DEBUG (ScheduledTrigger-28-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 6
   [junit4]   2> 11229 DEBUG (simCloudManagerPool-15-thread-23) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 11230 DEBUG (simCloudManagerPool-15-thread-23) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
19
   [junit4]   2> 11230 DEBUG (simCloudManagerPool-15-thread-23) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10004_solr, 127.0.0.1:10006_solr, 127.0.0.1:10005_solr, 
127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
   [junit4]   2> 11230 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 19
   [junit4]   2> 11230 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 11230 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 19, lastZnodeVersion 19
   [junit4]   2> 11250 DEBUG (ScheduledTrigger-28-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 6
   [junit4]   2> 11270 DEBUG (ScheduledTrigger-28-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 6
   [junit4]   2> 11291 DEBUG (ScheduledTrigger-28-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 6
   [junit4]   2> 11312 DEBUG (ScheduledTrigger-28-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 6
   [junit4]   2> 11328 DEBUG (ScheduledTrigger-28-thread-2) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"3bc9806e1f5f0Tnaka5ov337l12fifkg2hnna4",
   [junit4]   2>   "source":"search_rate_trigger",
   [junit4]   2>   "eventTime":1051786066654704,
   [junit4]   2>   "eventType":"SEARCHRATE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "node":{
   [junit4]   2>       "127.0.0.1:10001_solr":250.0,
   [junit4]   2>       "127.0.0.1:10006_solr":250.0},
   [junit4]   2>     "replica":[
   [junit4]   2>       "{\"core_node1\":{\n    
\"core\":\"collection1_shard1_replica_n1\",\n    
\"INDEX.sizeInBytes\":123450000,\n    \"rate\":250.0,\n    
\"node_name\":\"127.0.0.1:10001_solr\",\n    \"state\":\"active\",\n    
\"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250.0,\n    
\"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}",
   [junit4]   2>       "{\"core_node2\":{\n    
\"core\":\"collection1_shard1_replica_n2\",\n    
\"INDEX.sizeInBytes\":123450000,\n    \"rate\":250.0,\n    
\"node_name\":\"127.0.0.1:10006_solr\",\n    \"state\":\"active\",\n    
\"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250.0,\n    
\"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}"],
   [junit4]   2>     "collection":{"collection1":500.0},
   [junit4]   2>     "shard":{"collection1":{"shard1":500.0}}}}
   [junit4]   2> 11331 DEBUG (AutoscalingActionExecutor-29-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"3bc9806e1f5f0Tnaka5ov337l12fifkg2hnna4",
   [junit4]   2>   "source":"search_rate_trigger",
   [junit4]   2>   "eventTime":1051786066654704,
   [junit4]   2>   "eventType":"SEARCHRATE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "node":{
   [junit4]   2>       "127.0.0.1:10001_solr":250.0,
   [junit4]   2>       "127.0.0.1:10006_solr":250.0},
   [junit4]   2>     "replica":[
   [junit4]   2>       "{\"core_node1\":{\n    
\"core\":\"collection1_shard1_replica_n1\",\n    
\"INDEX.sizeInBytes\":123450000,\n    \"rate\":250.0,\n    
\"node_name\":\"127.0.0.1:10001_solr\",\n    \"state\":\"active\",\n    
\"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250.0,\n    
\"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}",
   

[...truncated too long message...]

xecutor-4594-thread-1, state=WAITING, group=TGRP-TestLazyCores]
   [junit4]   2> Dec 15, 2017 3:23:16 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be 
terminated:
   [junit4]   2>    1) Thread[id=10489, name=searcherExecutor-4594-thread-1, 
state=WAITING, group=TGRP-TestLazyCores]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2>         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2>         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2>         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   [junit4]   2>         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]   2>         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70), 
sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@31187953),
 locale=es-BO, timezone=Canada/Atlantic
   [junit4]   2> NOTE: Windows 10 10.0 amd64/Oracle Corporation 1.8.0_144 
(64-bit)/cpus=3,threads=2,free=196523056,total=502792192
   [junit4]   2> NOTE: All tests run in this JVM: 
[FieldAnalysisRequestHandlerTest, TestTriggerIntegration, TestPushWriter, 
DirectoryFactoryTest, CreateCollectionCleanupTest, 
TestSolrCloudWithKerberosAlt, TestWriterPerf, TestFieldCacheWithThreads, 
TestIndexingPerformance, CursorPagingTest, TestPhraseSuggestions, 
TestPHPSerializedResponseWriter, MultiThreadedOCPTest, 
ResponseLogComponentTest, CSVRequestHandlerTest, UnloadDistributedZkTest, 
AddReplicaTest, TestSubQueryTransformer, TestReqParamsAPI, TestPolicyCloud, 
TestPayloadCheckQParserPlugin, SpatialFilterTest, TestManagedStopFilterFactory, 
ZkCLITest, TestManagedSchemaAPI, TestJsonFacetRefinement, 
BasicAuthStandaloneTest, URLClassifyProcessorTest, StatsReloadRaceTest, 
TestSystemCollAutoCreate, TestUnifiedSolrHighlighter, ScriptEngineTest, 
TestFilteredDocIdSet, TestAddFieldRealTimeGet, CleanupOldIndexTest, 
CoreAdminCreateDiscoverTest, CollectionReloadTest, ShardRoutingTest, 
TestRequestForwarding, TestBackupRepositoryFactory, 
TestSerializedLuceneMatchVersion, TestSolrCloudWithHadoopAuthPlugin, 
DeleteNodeTest, DeleteInactiveReplicaTest, SearchRateTriggerTest, 
TestExactSharedStatsCache, HdfsDirectoryTest, PreAnalyzedUpdateProcessorTest, 
MultiTermTest, SignatureUpdateProcessorFactoryTest, 
WordBreakSolrSpellCheckerTest, DataDrivenBlockJoinTest, 
TestManagedResourceStorage, SolrRequestParserTest, TestClusterStateProvider, 
ResourceLoaderTest, TestQuerySenderListener, TestRecoveryHdfs, UUIDFieldTest, 
ManagedSchemaRoundRobinCloudTest, 
IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestSolrCoreProperties, 
TestFuzzyAnalyzedSuggestions, SuggesterWFSTTest, SolrTestCaseJ4Test, 
DirectUpdateHandlerTest, TestManagedSynonymGraphFilterFactory, 
TestRawTransformer, TestStressCloudBlindAtomicUpdates, 
AssignBackwardCompatibilityTest, TestTolerantSearch, TestIBSimilarityFactory, 
TestScoreJoinQPNoScore, TestCoreAdminApis, TestCoreDiscovery, 
DateMathParserTest, TestCloudRecovery, RequestHandlersTest, ImplicitSnitchTest, 
TestDefaultStatsCache, TestAuthorizationFramework, 
LeaderFailoverAfterPartitionTest, CurrencyFieldTypeTest, 
PathHierarchyTokenizerFactoryTest, TestTolerantUpdateProcessorCloud, 
TimeRoutedAliasUpdateProcessorTest, TestHashQParserPlugin, SearchHandlerTest, 
ClusterStateUpdateTest, DirectSolrConnectionTest, 
AutoAddReplicasPlanActionTest, TestReloadAndDeleteDocs, TestRandomDVFaceting, 
TestBulkSchemaAPI, StatsComponentTest, RestartWhileUpdatingTest, 
SuggesterFSTTest, SolrCmdDistributorTest, TestCloudNestedDocsSort, 
TestSkipOverseerOperations, ZkControllerTest, TestLMDirichletSimilarityFactory, 
TestLegacyFieldReuse, TestReplicationHandler, TestRebalanceLeaders, 
SuggesterTSTTest, TimeZoneUtilsTest, TestLazyCores]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLazyCores 
-Dtests.seed=9E68C9EAEE2F34B1 -Dtests.slow=true -Dtests.locale=es-BO 
-Dtests.timezone=Canada/Atlantic -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J0 | TestLazyCores (suite) <<<
   [junit4]    > Throwable #1: 
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE 
scope at org.apache.solr.core.TestLazyCores: 
   [junit4]    >    1) Thread[id=10489, name=searcherExecutor-4594-thread-1, 
state=WAITING, group=TGRP-TestLazyCores]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([9E68C9EAEE2F34B1]:0)Throwable #2: 
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=10489, name=searcherExecutor-4594-thread-1, 
state=WAITING, group=TGRP-TestLazyCores]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([9E68C9EAEE2F34B1]:0)
   [junit4] Completed [198/764 (3!)] on J0 in 32.54s, 9 tests, 3 errors <<< 
FAILURES!

[...truncated 7246 lines...]
   [junit4] Suite: org.apache.solr.handler.dataimport.TestTemplateTransformer
   [junit4]   2> Creating dataDir: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\contrib\solr-dataimporthandler\test\J1\temp\solr.handler.dataimport.TestTemplateTransformer_7829E85B5E0366EC-001\init-core-data-001
   [junit4]   2> 17753 T83 oas.SolrTestCaseJ4.randomizeNumericTypesProperties 
Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 17756 T83 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl 
(false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, 
value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 17756 T83 
oas.SolrTestCaseJ4.assertNonBlockingRandomGeneratorAvailable SecureRandom 
sanity checks: test.solr.allowed.securerandom=null & 
java.security.egd=file:/dev/./urandom
   [junit4]   2> 17758 T83 oas.SolrTestCaseJ4.setUp ###Starting testTransformRow
   [junit4]   2> 17767 T83 oas.SolrTestCaseJ4.tearDown ###Ending 
testTransformRow
   [junit4]   2> 17769 T83 oas.SolrTestCaseJ4.setUp ###Starting 
testTransformRowMultiValue
   [junit4]   2> 17770 T83 oas.SolrTestCaseJ4.tearDown ###Ending 
testTransformRowMultiValue
   [junit4]   2> 17772 T83 oas.SolrTestCaseJ4.clearObjectTrackerAndCheckEmpty 
------------------------------------------------------- Done waiting for 
tracked resources to be released
   [junit4]   2> NOTE: test params are: codec=Lucene70, 
sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@b487db2),
 locale=en-IN, timezone=Europe/Helsinki
   [junit4]   2> NOTE: Windows 10 10.0 amd64/Oracle Corporation 1.8.0_144 
(64-bit)/cpus=3,threads=1,free=125923240,total=184549376
   [junit4]   2> NOTE: All tests run in this JVM: [TestFieldReader, 
TestEphemeralCache, TestSolrEntityProcessorUnit, TestXPathRecordReader, 
TestXPathEntityProcessor, TestVariableResolver, TestTemplateTransformer]
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestTemplateTransformer -Dtests.seed=7829E85B5E0366EC 
-Dtests.slow=true -Dtests.locale=en-IN -Dtests.timezone=Europe/Helsinki 
-Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J1 | TestTemplateTransformer (suite) <<<
   [junit4]    > Throwable #1: java.io.IOException: Could not remove the 
following files (in the order of attempts):
   [junit4]    >    
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\contrib\solr-dataimporthandler\test\J1\temp\solr.handler.dataimport.TestTemplateTransformer_7829E85B5E0366EC-001\dih-properties-001:
 java.nio.file.AccessDeniedException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\contrib\solr-dataimporthandler\test\J1\temp\solr.handler.dataimport.TestTemplateTransformer_7829E85B5E0366EC-001\dih-properties-001
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([7829E85B5E0366EC]:0)
   [junit4]    >        at org.apache.lucene.util.IOUtils.rm(IOUtils.java:329)
   [junit4]    >        at java.lang.Thread.run(Thread.java:748)
   [junit4] Completed [7/38 (1!)] on J1 in 0.09s, 2 tests, 1 error <<< FAILURES!

[...truncated 41904 lines...]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to