Tyler Hobbs created CASSANDRA-7135:
--------------------------------------
Summary: Better checks and error messages for data and log dirs on
startup
Key: CASSANDRA-7135
URL: https://issues.apache.org/jira/browse/CASSANDRA-7135
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Tyler Hobbs
Fix For: 2.0.8
We used to have checks to make sure the data and logging dirs existed and had
good permissions on startup. In 2.0, it seems that's not the case any more.
If {{/var/lib/cassandra}} and {{/var/log/cassandra}} don't exist, you'll get a
startup log like this:
{noformat}
~/cassandra $ bin/cassandra -f
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /var/log/cassandra/system.log (No such file or
directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
at java.io.FileOutputStream.<init>(FileOutputStream.java:142)
at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
at
org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207)
at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
at
org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
at
org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
at
org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
at
org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809)
at
org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735)
at
org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615)
at
org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:502)
at
org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:395)
at
org.apache.log4j.PropertyWatchdog.doOnChange(PropertyConfigurator.java:922)
at
org.apache.log4j.helpers.FileWatchdog.checkAndConfigure(FileWatchdog.java:89)
at org.apache.log4j.helpers.FileWatchdog.<init>(FileWatchdog.java:58)
at
org.apache.log4j.PropertyWatchdog.<init>(PropertyConfigurator.java:914)
at
org.apache.log4j.PropertyConfigurator.configureAndWatch(PropertyConfigurator.java:461)
at
org.apache.cassandra.service.CassandraDaemon.initLog4j(CassandraDaemon.java:133)
at
org.apache.cassandra.service.CassandraDaemon.<clinit>(CassandraDaemon.java:81)
INFO 18:10:23,635 Logging initialized
INFO 18:10:23,670 Loading settings from
file:/home/thobbs/cassandra/conf/cassandra.yaml
INFO 18:10:24,061 Data files directories: [/var/lib/cassandra/data]
INFO 18:10:24,062 Commit log directory: /var/lib/cassandra/commitlog
INFO 18:10:24,062 DiskAccessMode 'auto' determined to be mmap, indexAccessMode
is mmap
INFO 18:10:24,062 disk_failure_policy is stop
INFO 18:10:24,062 commit_failure_policy is stop
INFO 18:10:24,066 Global memtable threshold is enabled at 981MB
INFO 18:10:24,186 Not using multi-threaded compaction
INFO 18:10:24,393 JVM vendor/version: Java HotSpot(TM) 64-Bit Server
VM/1.7.0_40
INFO 18:10:24,393 Heap size: 4116709376/4116709376
INFO 18:10:24,393 Code Cache Non-heap memory: init = 2555904(2496K) used =
664064(648K) committed = 2555904(2496K) max = 50331648(49152K)
INFO 18:10:24,393 Par Eden Space Heap memory: init = 671088640(655360K) used =
107378592(104861K) committed = 671088640(655360K) max = 671088640(655360K)
INFO 18:10:24,393 Par Survivor Space Heap memory: init = 83886080(81920K) used
= 0(0K) committed = 83886080(81920K) max = 83886080(81920K)
INFO 18:10:24,393 CMS Old Gen Heap memory: init = 3361734656(3282944K) used =
0(0K) committed = 3361734656(3282944K) max = 3361734656(3282944K)
INFO 18:10:24,394 CMS Perm Gen Non-heap memory: init = 21757952(21248K) used =
14180160(13847K) committed = 21757952(21248K) max = 85983232(83968K)
INFO 18:10:24,394 Classpath:
bin/../conf:bin/../build/classes/main:bin/../build/classes/thrift:bin/../lib/antlr-3.2.jar:bin/../lib/commons-cli-1.1.jar:bin/../lib/commons-codec-1.2.jar:bin/../lib/commons-lang3-3.1.jar:bin/../lib/compress-lzf-0.8.4.jar:bin/../lib/concurrentlinkedhashmap-lru-1.3.jar:bin/../lib/disruptor-3.0.1.jar:bin/../lib/guava-15.0.jar:bin/../lib/high-scale-lib-1.1.2.jar:bin/../lib/jackson-core-asl-1.9.2.jar:bin/../lib/jackson-mapper-asl-1.9.2.jar:bin/../lib/jamm-0.2.5.jar:bin/../lib/jbcrypt-0.3m.jar:bin/../lib/jline-1.0.jar:bin/../lib/json-simple-1.1.jar:bin/../lib/libthrift-0.9.1.jar:bin/../lib/log4j-1.2.16.jar:bin/../lib/lz4-1.2.0.jar:bin/../lib/metrics-core-2.2.0.jar:bin/../lib/netty-3.6.6.Final.jar:bin/../lib/reporter-config-2.1.0.jar:bin/../lib/servlet-api-2.5-20081211.jar:bin/../lib/slf4j-api-1.7.2.jar:bin/../lib/slf4j-log4j12-1.7.2.jar:bin/../lib/snakeyaml-1.11.jar:bin/../lib/snappy-java-1.0.5.jar:bin/../lib/snaptree-0.1.jar:bin/../lib/super-csv-2.1.0.jar:bin/../lib/thrift-server-0.3.3.jar:bin/../lib/jamm-0.2.5.jar
INFO 18:10:24,395 JNA not found. Native methods will be disabled.
INFO 18:10:24,410 Initializing key cache with capacity of 100 MBs.
INFO 18:10:24,423 Scheduling key cache save to each 14400 seconds (going to
save all keys).
INFO 18:10:24,424 Initializing row cache with capacity of 0 MBs
INFO 18:10:24,433 Scheduling row cache save to each 0 seconds (going to save
all keys).
ERROR 18:10:24,461 Failed to create
/var/lib/cassandra/data/system/schema_triggers directory
ERROR 18:10:24,468 Failed to create
/var/lib/cassandra/data/system/schema_triggers directory
ERROR 18:10:24,469 Failed to create
/var/lib/cassandra/data/system/schema_triggers directory
ERROR 18:10:24,469 Failed to create
/var/lib/cassandra/data/system/schema_triggers directory
ERROR 18:10:24,470 Failed to create
/var/lib/cassandra/data/system/schema_triggers directory
ERROR 18:10:24,470 Failed to create
/var/lib/cassandra/data/system/compaction_history directory
ERROR 18:10:24,471 Failed to create
/var/lib/cassandra/data/system/compaction_history directory
ERROR 18:10:24,471 Failed to create
/var/lib/cassandra/data/system/compaction_history directory
ERROR 18:10:24,472 Failed to create
/var/lib/cassandra/data/system/compaction_history directory
ERROR 18:10:24,472 Failed to create
/var/lib/cassandra/data/system/compaction_history directory
ERROR 18:10:24,473 Failed to create
/var/lib/cassandra/data/system/compaction_history directory
ERROR 18:10:24,474 Failed to create
/var/lib/cassandra/data/system/compaction_history directory
ERROR 18:10:24,474 Failed to create
/var/lib/cassandra/data/system/compaction_history directory
ERROR 18:10:24,475 Failed to create /var/lib/cassandra/data/system/batchlog
directory
ERROR 18:10:24,475 Failed to create /var/lib/cassandra/data/system/batchlog
directory
ERROR 18:10:24,476 Failed to create /var/lib/cassandra/data/system/batchlog
directory
ERROR 18:10:24,476 Failed to create /var/lib/cassandra/data/system/batchlog
directory
ERROR 18:10:24,477 Failed to create /var/lib/cassandra/data/system/batchlog
directory
ERROR 18:10:24,477 Failed to create
/var/lib/cassandra/data/system/sstable_activity directory
ERROR 18:10:24,479 Failed to create
/var/lib/cassandra/data/system/sstable_activity directory
ERROR 18:10:24,480 Failed to create
/var/lib/cassandra/data/system/sstable_activity directory
ERROR 18:10:24,480 Failed to create
/var/lib/cassandra/data/system/sstable_activity directory
ERROR 18:10:24,481 Failed to create
/var/lib/cassandra/data/system/sstable_activity directory
ERROR 18:10:24,481 Failed to create
/var/lib/cassandra/data/system/sstable_activity directory
ERROR 18:10:24,482 Failed to create /var/lib/cassandra/data/system/peer_events
directory
ERROR 18:10:24,482 Failed to create /var/lib/cassandra/data/system/peer_events
directory
ERROR 18:10:24,483 Failed to create /var/lib/cassandra/data/system/peer_events
directory
ERROR 18:10:24,484 Failed to create
/var/lib/cassandra/data/system/compactions_in_progress directory
ERROR 18:10:24,484 Failed to create
/var/lib/cassandra/data/system/compactions_in_progress directory
ERROR 18:10:24,485 Failed to create
/var/lib/cassandra/data/system/compactions_in_progress directory
ERROR 18:10:24,485 Failed to create
/var/lib/cassandra/data/system/compactions_in_progress directory
ERROR 18:10:24,486 Failed to create
/var/lib/cassandra/data/system/compactions_in_progress directory
ERROR 18:10:24,486 Failed to create /var/lib/cassandra/data/system/hints
directory
ERROR 18:10:24,487 Failed to create /var/lib/cassandra/data/system/hints
directory
ERROR 18:10:24,488 Failed to create /var/lib/cassandra/data/system/hints
directory
ERROR 18:10:24,488 Failed to create /var/lib/cassandra/data/system/hints
directory
ERROR 18:10:24,489 Failed to create /var/lib/cassandra/data/system/hints
directory
ERROR 18:10:24,489 Failed to create
/var/lib/cassandra/data/system/schema_keyspaces directory
ERROR 18:10:24,490 Failed to create
/var/lib/cassandra/data/system/schema_keyspaces directory
ERROR 18:10:24,490 Failed to create
/var/lib/cassandra/data/system/schema_keyspaces directory
ERROR 18:10:24,491 Failed to create
/var/lib/cassandra/data/system/schema_keyspaces directory
ERROR 18:10:24,491 Failed to create
/var/lib/cassandra/data/system/schema_keyspaces directory
ERROR 18:10:24,492 Failed to create /var/lib/cassandra/data/system/range_xfers
directory
ERROR 18:10:24,492 Failed to create /var/lib/cassandra/data/system/range_xfers
directory
ERROR 18:10:24,493 Failed to create /var/lib/cassandra/data/system/range_xfers
directory
ERROR 18:10:24,494 Failed to create
/var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,494 Failed to create
/var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,495 Failed to create
/var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,495 Failed to create
/var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,496 Failed to create
/var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,496 Failed to create
/var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,497 Failed to create
/var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,498 Failed to create
/var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,498 Failed to create
/var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,499 Failed to create
/var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,499 Failed to create
/var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,500 Failed to create
/var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,500 Failed to create
/var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,501 Failed to create
/var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,502 Failed to create
/var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,502 Failed to create
/var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,503 Failed to create
/var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,504 Failed to create
/var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,504 Failed to create
/var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,505 Failed to create
/var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,506 Failed to create
/var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,506 Failed to create
/var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,507 Failed to create
/var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,507 Failed to create
/var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,508 Failed to create
/var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,508 Failed to create
/var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,509 Failed to create
/var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,510 Failed to create
/var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,510 Failed to create
/var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,511 Failed to create /var/lib/cassandra/data/system/NodeIdInfo
directory
ERROR 18:10:24,511 Failed to create /var/lib/cassandra/data/system/NodeIdInfo
directory
ERROR 18:10:24,512 Failed to create /var/lib/cassandra/data/system/NodeIdInfo
directory
ERROR 18:10:24,512 Failed to create /var/lib/cassandra/data/system/NodeIdInfo
directory
ERROR 18:10:24,513 Failed to create /var/lib/cassandra/data/system/paxos
directory
ERROR 18:10:24,513 Failed to create /var/lib/cassandra/data/system/paxos
directory
ERROR 18:10:24,514 Failed to create /var/lib/cassandra/data/system/paxos
directory
ERROR 18:10:24,514 Failed to create /var/lib/cassandra/data/system/paxos
directory
ERROR 18:10:24,515 Failed to create /var/lib/cassandra/data/system/paxos
directory
ERROR 18:10:24,515 Failed to create /var/lib/cassandra/data/system/paxos
directory
ERROR 18:10:24,516 Failed to create /var/lib/cassandra/data/system/paxos
directory
ERROR 18:10:24,516 Failed to create /var/lib/cassandra/data/system/paxos
directory
ERROR 18:10:24,517 Failed to create
/var/lib/cassandra/data/system/schema_columns directory
ERROR 18:10:24,517 Failed to create
/var/lib/cassandra/data/system/schema_columns directory
ERROR 18:10:24,518 Failed to create
/var/lib/cassandra/data/system/schema_columns directory
ERROR 18:10:24,519 Failed to create
/var/lib/cassandra/data/system/schema_columns directory
ERROR 18:10:24,519 Failed to create
/var/lib/cassandra/data/system/schema_columns directory
ERROR 18:10:24,520 Failed to create
/var/lib/cassandra/data/system/schema_columns directory
ERROR 18:10:24,520 Failed to create
/var/lib/cassandra/data/system/schema_columns directory
ERROR 18:10:24,521 Failed to create
/var/lib/cassandra/data/system/schema_columns directory
ERROR 18:10:24,521 Failed to create
/var/lib/cassandra/data/system/schema_columns directory
ERROR 18:10:24,521 Failed to create
/var/lib/cassandra/data/system/schema_columns directory
ERROR 18:10:24,522 Failed to create /var/lib/cassandra/data/system/IndexInfo
directory
ERROR 18:10:24,522 Failed to create /var/lib/cassandra/data/system/IndexInfo
directory
ERROR 18:10:24,522 Failed to create /var/lib/cassandra/data/system/IndexInfo
directory
ERROR 18:10:24,523 Failed to create /var/lib/cassandra/data/system/IndexInfo
directory
ERROR 18:10:24,523 Failed to create /var/lib/cassandra/data/system/peers
directory
ERROR 18:10:24,523 Failed to create /var/lib/cassandra/data/system/peers
directory
ERROR 18:10:24,524 Failed to create /var/lib/cassandra/data/system/peers
directory
ERROR 18:10:24,524 Failed to create /var/lib/cassandra/data/system/peers
directory
ERROR 18:10:24,524 Failed to create /var/lib/cassandra/data/system/peers
directory
ERROR 18:10:24,525 Failed to create /var/lib/cassandra/data/system/peers
directory
ERROR 18:10:24,525 Failed to create /var/lib/cassandra/data/system/peers
directory
ERROR 18:10:24,526 Failed to create /var/lib/cassandra/data/system/peers
directory
ERROR 18:10:24,526 Failed to create /var/lib/cassandra/data/system/peers
directory
ERROR 18:10:24,526 Failed to create /var/lib/cassandra/data/system/peers
directory
ERROR 18:10:24,527 Failed to create /var/lib/cassandra/data/system/local
directory
ERROR 18:10:24,527 Failed to create /var/lib/cassandra/data/system/local
directory
ERROR 18:10:24,527 Failed to create /var/lib/cassandra/data/system/local
directory
ERROR 18:10:24,528 Failed to create /var/lib/cassandra/data/system/local
directory
ERROR 18:10:24,528 Failed to create /var/lib/cassandra/data/system/local
directory
ERROR 18:10:24,528 Failed to create /var/lib/cassandra/data/system/local
directory
ERROR 18:10:24,529 Failed to create /var/lib/cassandra/data/system/local
directory
ERROR 18:10:24,529 Failed to create /var/lib/cassandra/data/system/local
directory
ERROR 18:10:24,530 Failed to create /var/lib/cassandra/data/system/local
directory
ERROR 18:10:24,530 Failed to create /var/lib/cassandra/data/system/local
directory
ERROR 18:10:24,530 Failed to create /var/lib/cassandra/data/system/local
directory
ERROR 18:10:24,531 Failed to create /var/lib/cassandra/data/system/local
directory
ERROR 18:10:24,531 Failed to create /var/lib/cassandra/data/system/local
directory
ERROR 18:10:24,531 Failed to create /var/lib/cassandra/data/system/local
directory
ERROR 18:10:24,532 Failed to create /var/lib/cassandra/data/system/local
directory
ERROR 18:10:24,532 Failed to create /var/lib/cassandra/data/system/local
directory
ERROR 18:10:24,535 Fatal error: java.io.IOException: Failed to mkdirs
/var/lib/cassandra/data
Failed to mkdirs /var/lib/cassandra/data; unable to start server
{noformat}
To summarize, we need to:
* Make sure the directories exist
** If not, suggest creating them or editing cassandra.yaml
* Make sure the directories have good permissions
** If not, say what permissions are needed and link to instructions for fixing
them
--
This message was sent by Atlassian JIRA
(v6.2#6252)
