Hello Joseph, Your datanodes are in a bad state, you probably overwhelmed it when indexing. Check your max open files on those nodes. Usual default of 1024 is way too low.
Markus -----Original message----- > From:Joseph Obernberger <joseph.obernber...@gmail.com> > Sent: Monday 20th June 2016 19:36 > To: solr-user@lucene.apache.org > Subject: All Datanodes are Bad > > Anyone ever seen an error like this? We are running using HDFS for the > index. At the time of the error, we are doing a lot of indexing. > > Two errors: > java.io.IOException: All datanodes DatanodeInfoWithStorage[ > 172.16.100.220:50010,DS-4b806395-0661-4a70-a32b-deef82a85359,DISK] are bad. > Aborting... > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1357) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1119) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:622) > > and > > auto commit error...:org.apache.solr.common.SolrException: > java.io.IOException: All datanodes DatanodeInfoWithStorage[ > 172.16.100.220:50010,DS-4b806395-0661-4a70-a32b-deef82a85359,DISK] are bad. > Aborting... > at > org.apache.solr.update.HdfsTransactionLog.close(HdfsTransactionLog.java:321) > at org.apache.solr.update.TransactionLog.decref(TransactionLog.java:510) > at org.apache.solr.update.UpdateLog.addOldLog(UpdateLog.java:372) > at org.apache.solr.update.UpdateLog.postCommit(UpdateLog.java:668) > at > org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:658) > at org.apache.solr.update.CommitTracker.run(CommitTracker.java:217) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.io.IOException: All datanodes DatanodeInfoWithStorage[ > 172.16.100.220:50010,DS-4b806395-0661-4a70-a32b-deef82a85359,DISK] are bad. > Aborting... > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1357) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1119) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:622) > > On the client side doing the indexing, we see: > > org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error > from server at http://deimos:9100/solr/UNCLASS: java.io.IOException: All > datanodes > DatanodeInfoWithStorage[172.16.100.220:50010,DS-f0e14105-9557-4a59-8918-43724aaa8346,DISK] > are bad. Aborting... > at > org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:632) > at > org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:981) > at > org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:870) > at > org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:806) > at > org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149) > at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:106) > at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:71) > at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:85) > . > . > . > Caused by: > org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error > from server at http://deimos:9100/solr/UNCLASS: java.io.IOException: All > datanodes > DatanodeInfoWithStorage[172.16.100.220:50010,DS-f0e14105-9557-4a59-8918-43724aaa8346,DISK] > are bad. Aborting... > at > org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:576) > at > org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:240) > at > org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:229) > at > org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:372) > at > org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:325) > at > org.apache.solr.client.solrj.impl.CloudSolrClient$2.call(CloudSolrClient.java:607) > at > org.apache.solr.client.solrj.impl.CloudSolrClient$2.call(CloudSolrClient.java:604) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231) > > Thanks! > > -Joe >