Hi All
We are trying to High Availability solution for our database. We have
our application running on a node-1 and the DB running on the shared
storage which is shared across node-1 & node-2. Whenever there is a
power Failure on node-1 , we start our application on node-2 but our
problem is that now after a failover onto node-2 the DB deamon on node-2
will fail to start.
Mysql version information:
******************************************
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 58
Server version: 5.6.10-enterprise-commercial-advanced MySQL Enterprise
Server - Advanced Edition (Commercial)
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input
statement.
mysql>
*******************************************
On the node-2 when the DB deamon tries to start we are seeing the
following error messages in the log file:
Error Message *************************************************************
150703 16:01:11 mysqld_safe Starting mysqld daemon with databases from
/var/lib/mysql/data
2015-07-03 16:01:11 0 [Warning] TIMESTAMP with implicit DEFAULT value is
deprecated. Please use --explicit_defaults_for_timestamp server option
(see documentation for more details).
2015-07-03 16:01:11 16278 [Note] Plugin 'FEDERATED' is disabled.
2015-07-03 16:01:12 16278 [Note] InnoDB: The InnoDB memory heap is disabled
2015-07-03 16:01:12 16278 [Note] InnoDB: Mutexes and rw_locks use GCC
atomic builtins
2015-07-03 16:01:12 16278 [Note] InnoDB: Compressed tables use zlib 1.2.3
2015-07-03 16:01:12 16278 [Note] InnoDB: CPU does not support crc32
instructions
2015-07-03 16:01:12 16278 [Note] InnoDB: Using Linux native AIO
2015-07-03 16:01:12 16278 [Note] InnoDB: Initializing buffer pool, size
= 1.0G
2015-07-03 16:01:12 16278 [Note] InnoDB: Completed initialization of
buffer pool
2015-07-03 16:01:12 16278 [ERROR] InnoDB: Unable to lock ./ibdata1,
error: 11
2015-07-03 16:01:12 16278 [Note] InnoDB: Check that you do not already
have another mysqld process using the same InnoDB data or log files.
2015-07-03 16:01:12 16278 [Note] InnoDB: Retrying to lock the first data
file
2015-07-03 16:01:13 16278 [ERROR] InnoDB: Unable to lock ./ibdata1,
error: 11
2015-07-03 16:01:13 16278 [Note] InnoDB: Check that you do not already
have another mysqld process using the same InnoDB data or log files.
2015-07-03 16:01:14 16278 [ERROR] InnoDB: Unable to lock ./ibdata1,
error: 11
2015-07-03 16:01:14 16278 [Note] InnoDB: Check that you do not already
have another mysqld process using the same InnoDB data or log files.
2015-07-03 16:01:15 16278 [ERROR] InnoDB: Unable to lock ./ibdata1,
error: 11
2015-07-03 16:02:52 16278 [Note] InnoDB: Check that you do not already
have another mysqld process using the same InnoDB data or log files.
2015-07-03 16:02:52 16278 [Note] InnoDB: Unable to open the first data file
2015-07-03 16:02:52 7f9b9194e720 InnoDB: Operating system error number
11 in a file operation.
InnoDB: Error number 11 means 'Resource temporarily unavailable'.
InnoDB: Some operating system error numbers are described at
InnoDB:
http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html
2015-07-03 16:02:52 16278 [ERROR] InnoDB: Can't open './ibdata1'
2015-07-03 16:02:52 16278 [ERROR] InnoDB: Could not open or create the
system tablespace. If you tried to add new data files to the system
tablespace, and it failed here, you should now edit
innodb_data_file_path in my.cnf back to what it was, and remove the new
ibdata files InnoDB created in this failed attempt. InnoDB only wrote
those files full of zeros, but did not yet use them in any way. But be
careful: do not remove old data files which contain your precious data!
2015-07-03 16:02:52 16278 [ERROR] Plugin 'InnoDB' init function returned
error.
2015-07-03 16:02:52 16278 [ERROR] Plugin 'InnoDB' registration as a
STORAGE ENGINE failed.
2015-07-03 16:02:52 16278 [ERROR] Unknown/unsupported storage engine: InnoDB
2015-07-03 16:02:52 16278 [ERROR] Aborting
2015-07-03 16:02:52 16278 [Note] Binlog end
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'partition'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'ARCHIVE'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'BLACKHOLE'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin
'INNODB_SYS_TABLESPACES'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin
'INNODB_SYS_FOREIGN_COLS'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin
'INNODB_SYS_TABLESTATS'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin
'INNODB_FT_INDEX_TABLE'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin
'INNODB_FT_INDEX_CACHE'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin
'INNODB_FT_BEING_DELETED'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_FT_INSERTED'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin
'INNODB_FT_DEFAULT_STOPWORD'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_METRICS'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin
'INNODB_BUFFER_POOL_STATS'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin
'INNODB_BUFFER_PAGE_LRU'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin
'INNODB_CMP_PER_INDEX_RESET'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_CMPMEM'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_CMP'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_LOCKS'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_TRX'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'CSV'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'MEMORY'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'MRG_MYISAM'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'MyISAM'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'sha256_password'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'mysql_old_password'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin
'mysql_native_password'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'binlog'
2015-07-03 16:02:52 16278 [Note] /var/lib/mysql/bin/mysqld: Shutdown
complete
150703 16:02:52 mysqld_safe mysqld from pid file
/var/lib/mysql/mysqld.pid ended
150703 16:04:02 mysqld_safe Starting mysqld daemon with databases from
/var/lib/mysql/data
2015-07-03 16:04:03 0 [Warning] TIMESTAMP with implicit DEFAULT value is
deprecated. Please use --explicit_defaults_for_timestamp server option
(see documentation for more details).
2015-07-03 16:04:03 17753 [Note] Plugin 'FEDERATED' is disabled.
2015-07-03 16:04:03 17753 [Note] InnoDB: The InnoDB memory heap is disabled
2015-07-03 16:04:03 17753 [Note] InnoDB: Mutexes and rw_locks use GCC
atomic builtins
2015-07-03 16:04:03 17753 [Note] InnoDB: Compressed tables use zlib 1.2.3
2015-07-03 16:04:03 17753 [Note] InnoDB: CPU does not support crc32
instructions
2015-07-03 16:04:03 17753 [Note] InnoDB: Using Linux native AIO
2015-07-03 16:04:03 17753 [Note] InnoDB: Initializing buffer pool, size
= 1.0G
2015-07-03 16:04:03 17753 [Note] InnoDB: Completed initialization of
buffer pool
2015-07-03 16:04:03 17753 [ERROR] InnoDB: Unable to lock ./ibdata1,
error: 11
2015-07-03 16:04:03 17753 [Note] InnoDB: Check that you do not already
have another mysqld process using the same InnoDB data or log files.
2015-07-03 16:04:03 17753 [Note] InnoDB: Retrying to lock the first data
file
2015-07-03 16:04:04 17753 [ERROR] InnoDB: Unable to lock ./ibdata1,
error: 11
2015-07-03 16:04:04 17753 [Note] InnoDB: Check that you do not already
have another mysqld process using the same InnoDB data or log files.
2015-07-03 16:04:05 17753 [ERROR] InnoDB: Unable to lock ./ibdata1,
error: 11
2015-07-03 16:04:05 17753 [Note] InnoDB: Check that you do not already
have another mysqld process using the same InnoDB data or log files.
2015-07-03 16:04:06 17753 [ERROR] InnoDB: Unable to lock ./ibdata1,
error: 11
2015-07-03 16:04:06 17753 [Note] InnoDB: Check that you do not already
have another mysqld process using the same InnoDB data or log files.
2015-07-03 16:04:07 17753 [ERROR] InnoDB: Unable to lock ./ibdata1,
error: 11
2015-07-03 16:04:07 17753 [Note] InnoDB: Check that you do not already
have another mysqld process using the same InnoDB data or log files.
2015-07-03 16:04:08 17753 [ERROR] InnoDB: Unable to lock ./ibdata1,
error: 11
2015-07-03 16:05:43 17753 [Note] InnoDB: Check that you do not already
have another mysqld process using the same InnoDB data or log files.
2015-07-03 16:05:43 17753 [Note] InnoDB: Unable to open the first data file
2015-07-03 16:05:43 7f4cbe5a3720 InnoDB: Operating system error number
11 in a file operation.
InnoDB: Error number 11 means 'Resource temporarily unavailable'.
InnoDB: Some operating system error numbers are described at
InnoDB:
http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html
2015-07-03 16:05:43 17753 [ERROR] InnoDB: Can't open './ibdata1'
2015-07-03 16:05:43 17753 [ERROR] InnoDB: Could not open or create the
system tablespace. If you tried to add new data files to the system
tablespace, and it failed here, you should now edit
innodb_data_file_path in my.cnf back to what it was, and remove the new
ibdata files InnoDB created in this failed attempt. InnoDB only wrote
those files full of zeros, but did not yet use them in any way. But be
careful: do not remove old data files which contain your precious data!
2015-07-03 16:05:43 17753 [ERROR] Plugin 'InnoDB' init function returned
error.
2015-07-03 16:05:43 17753 [ERROR] Plugin 'InnoDB' registration as a
STORAGE ENGINE failed.
2015-07-03 16:05:43 17753 [ERROR] Unknown/unsupported storage engine: InnoDB
2015-07-03 16:05:43 17753 [ERROR] Aborting
2015-07-03 16:05:43 17753 [Note] Binlog end
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'partition'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'ARCHIVE'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'BLACKHOLE'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin
'INNODB_SYS_TABLESPACES'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin
'INNODB_SYS_FOREIGN_COLS'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin
'INNODB_SYS_TABLESTATS'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin
'INNODB_FT_INDEX_TABLE'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin
'INNODB_FT_INDEX_CACHE'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin
'INNODB_FT_BEING_DELETED'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_FT_INSERTED'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin
'INNODB_FT_DEFAULT_STOPWORD'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_METRICS'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin
'INNODB_BUFFER_POOL_STATS'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin
'INNODB_BUFFER_PAGE_LRU'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin
'INNODB_CMP_PER_INDEX_RESET'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_CMPMEM'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_CMP'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_LOCKS'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_TRX'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'CSV'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'MEMORY'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'MRG_MYISAM'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'MyISAM'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'sha256_password'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'mysql_old_password'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin
'mysql_native_password'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'binlog'
2015-07-03 16:05:43 17753 [Note] /var/lib/mysql/bin/mysqld: Shutdown
complete
150703 16:05:43 mysqld_safe mysqld from pid file
/var/lib/mysql/mysqld.pid ended
End of Error Message
*****************************************************************
Appreciate your help in this regard.
Thanks
Jatin
