Hi. Recently, there has occasionally been a stuck mini cluster test, usually MiniTez, I think I’ve seen MiniSpark too. Upon looking at the logs for one recent timeout, I don’t see any obvious problem with Hive, but there’s this:
2016-01-25T09:23:15,658 WARN [org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonito r@6845113[]]: namenode.FSNamesystem (FSNamesystem.java:run(5159)) - NameNode low on available disk space. Entering safe mode. 2016-01-25T09:23:15,658 INFO [org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonito r@6845113[]]: hdfs.StateChange (FSNamesystem.java:reportStatus(6003)) - STATE* Safe mode is ON. Resources are low on NN. Please add or free up more resources then turn off safe mode manually. NOTE: If you turn off safe mode before adding resources, the NN will immediately return to safe mode. Use "hdfs dfsadmin -safemode leave" to turn safe mode off. After that, there are many safe-mode related errors. I think that could be related to recent changes to preserve more logs. Can we perhaps scale down log preservation a little bit, or gzip them? It could also be something else on HiveQA machine.
