Re: Master shuts down during log splitting on restart
HBASE-12095 was logged by Ashish. I provided patch on that issue. FYI On Thu, Sep 25, 2014 at 1:45 AM, Ted Yu wrote: > Can you try the patch below ? > > http://pastebin.com/SnYZQf7Y > > Cheers > > On Wed, Sep 24, 2014 at 10:36 PM, ashish singhi > wrote: > >> Hi All. >> >> I am using 0.98.6 HBase. >> >> I observed that when I have the following value set in my hbase-site.xml >> file >> >> >> hbase.regionserver.wal.encryption >> false >> >> >> hbase.regionserver.hlog.reader.impl >> >> org.apache.hadoop.hbase.regionserver.wal.SecureProtobufLogReader >> >> >> hbase.regionserver.hlog.writer.impl >> >> org.apache.hadoop.hbase.regionserver.wal.SecureProtobufLogWriter >> >> >> And while log splitting on hbase service restart, master shutdown with >> following exception. >> >> Exception in master log >> >> 2014-09-24 17:14:28,590 FATAL [master:host-10-18-40-18:6] >> master.HMaster: Master server abort: loaded coprocessors are: >> [org.apache.hadoop.hbase.security.access.AccessController] >> 2014-09-24 17:14:28,590 FATAL [master:host-10-18-40-18:6] >> master.HMaster: Unhandled exception. Starting shutdown. >> java.io.IOException: error or interrupted while splitting logs in [hdfs:// >> 10.18.40.18:8020/tmp/hbase-ashish/hbase/WALs/host-10-18-40-18,60020,1411558717849-splitting] >> Task = installed = 6 done = 0 error = 6 >> at >> org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:378) >> at >> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:415) >> at >> org.apache.hadoop.hbase.master.MasterFileSystem.splitMetaLog(MasterFileSystem.java:307) >> at >> org.apache.hadoop.hbase.master.MasterFileSystem.splitMetaLog(MasterFileSystem.java:298) >> at >> org.apache.hadoop.hbase.master.HMaster.splitMetaLogBeforeAssignment(HMaster.java:1071) >> at >> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:863) >> at >> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:612) >> at java.lang.Thread.run(Thread.java:745) >> >> Exception in region server log >> >> 2014-09-24 20:10:16,535 WARN >> [RS_LOG_REPLAY_OPS-host-10-18-40-18:60020-1] regionserver.SplitLogWorker: >> log splitting of >> WALs/host-10-18-40-18,60020,1411558717849-splitting/host-10-18-40-18%2C60020%2C1411558717849.1411558724316.meta >> failed, returning error >> java.io.IOException: Cannot get log reader >> at >> org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:161) >> at >> org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:89) >> at >> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:660) >> at >> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:569) >> at >> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:282) >> at >> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:225) >> at >> org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:143) >> at >> org.apache.hadoop.hbase.regionserver.handler.HLogSplitterHandler.process(HLogSplitterHandler.java:82) >> at >> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:745) >> Caused by: java.lang.UnsupportedOperationException: Unable to find >> suitable constructor for class >> org.apache.hadoop.hbase.regionserver.wal.SecureWALCellCodec >> at >> org.apache.hadoop.hbase.util.ReflectionUtils.instantiateWithCustomCtor(ReflectionUtils.java:39) >> at >> org.apache.hadoop.hbase.regionserver.wal.WALCellCodec.create(WALCellCodec.java:101) >> at >> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.getCodec(ProtobufLogReader.java:242) >> at >> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initAfterCompression(ProtobufLogReader.java:247) >> at >> org.apache.hadoop.hbase.regionserver.wal.SecureProtobufLogReader.initAfterCompression(SecureProtobufLogReader.java:138) >> at >> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:85) >> at >> org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:113) >> ... 11 more >> Caused by: java.lang.NoSuchMethodException: >> org.apache.hadoop.hbase.regionserver.wal.SecureWALCellCodec.(org.apache.hadoop.conf.Configuration, >>
[jira] [Created] (HBASE-12095) Master shuts down during log splitting on restart
Ashish Singhi created HBASE-12095: - Summary: Master shuts down during log splitting on restart Key: HBASE-12095 URL: https://issues.apache.org/jira/browse/HBASE-12095 Project: HBase Issue Type: Bug Affects Versions: 0.98.6 Reporter: Ashish Singhi Fix For: 0.98.7 I observed that when I have the following value set in my hbase-site.xml file hbase.regionserver.wal.encryption false hbase.regionserver.hlog.reader.impl org.apache.hadoop.hbase.regionserver.wal.SecureProtobufLogReader hbase.regionserver.hlog.writer.impl org.apache.hadoop.hbase.regionserver.wal.SecureProtobufLogWriter And while log splitting on hbase service restart, master shutdown with following exception. Exception in master log 2014-09-24 17:14:28,590 FATAL [master:host-10-18-40-18:6] master.HMaster: Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.security.access.AccessController] 2014-09-24 17:14:28,590 FATAL [master:host-10-18-40-18:6] master.HMaster: Unhandled exception. Starting shutdown. java.io.IOException: error or interrupted while splitting logs in [hdfs://10.18.40.18:8020/tmp/hbase-ashish/hbase/WALs/host-10-18-40-18,60020,1411558717849-splitting] Task = installed = 6 done = 0 error = 6 at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:378) at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:415) at org.apache.hadoop.hbase.master.MasterFileSystem.splitMetaLog(MasterFileSystem.java:307) at org.apache.hadoop.hbase.master.MasterFileSystem.splitMetaLog(MasterFileSystem.java:298) at org.apache.hadoop.hbase.master.HMaster.splitMetaLogBeforeAssignment(HMaster.java:1071) at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:863) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:612) at java.lang.Thread.run(Thread.java:745) Exception in region server log 2014-09-24 20:10:16,535 WARN [RS_LOG_REPLAY_OPS-host-10-18-40-18:60020-1] regionserver.SplitLogWorker: log splitting of WALs/host-10-18-40-18,60020,1411558717849-splitting/host-10-18-40-18%2C60020%2C1411558717849.1411558724316.meta failed, returning error java.io.IOException: Cannot get log reader at org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:161) at org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:89) at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:660) at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:569) at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:282) at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:225) at org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:143) at org.apache.hadoop.hbase.regionserver.handler.HLogSplitterHandler.process(HLogSplitterHandler.java:82) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.UnsupportedOperationException: Unable to find suitable constructor for class org.apache.hadoop.hbase.regionserver.wal.SecureWALCellCodec at org.apache.hadoop.hbase.util.ReflectionUtils.instantiateWithCustomCtor(ReflectionUtils.java:39) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec.create(WALCellCodec.java:101) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.getCodec(ProtobufLogReader.java:242) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initAfterCompression(ProtobufLogReader.java:247) at org.apache.hadoop.hbase.regionserver.wal.SecureProtobufLogReader.initAfterCompression(SecureProtobufLogReader.java:138) at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:85) at org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:113) ... 11 more Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.hbase.regionserver.wal.SecureWALCellCodec.(org.apache.hadoop.conf.Configuration, org.apache.hadoop.hbase.regionserver.wal.CompressionContext) at java.lang.Class.getConstructor0(Class.java:2849) at
Re: Master shuts down during log splitting on restart
Can you try the patch below ? http://pastebin.com/SnYZQf7Y Cheers On Wed, Sep 24, 2014 at 10:36 PM, ashish singhi wrote: > Hi All. > > I am using 0.98.6 HBase. > > I observed that when I have the following value set in my hbase-site.xml > file > > > hbase.regionserver.wal.encryption > false > > > hbase.regionserver.hlog.reader.impl > > org.apache.hadoop.hbase.regionserver.wal.SecureProtobufLogReader > > > hbase.regionserver.hlog.writer.impl > > org.apache.hadoop.hbase.regionserver.wal.SecureProtobufLogWriter > > > And while log splitting on hbase service restart, master shutdown with > following exception. > > Exception in master log > > 2014-09-24 17:14:28,590 FATAL [master:host-10-18-40-18:6] > master.HMaster: Master server abort: loaded coprocessors are: > [org.apache.hadoop.hbase.security.access.AccessController] > 2014-09-24 17:14:28,590 FATAL [master:host-10-18-40-18:6] > master.HMaster: Unhandled exception. Starting shutdown. > java.io.IOException: error or interrupted while splitting logs in [hdfs:// > 10.18.40.18:8020/tmp/hbase-ashish/hbase/WALs/host-10-18-40-18,60020,1411558717849-splitting] > Task = installed = 6 done = 0 error = 6 > at > org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:378) > at > org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:415) > at > org.apache.hadoop.hbase.master.MasterFileSystem.splitMetaLog(MasterFileSystem.java:307) > at > org.apache.hadoop.hbase.master.MasterFileSystem.splitMetaLog(MasterFileSystem.java:298) > at > org.apache.hadoop.hbase.master.HMaster.splitMetaLogBeforeAssignment(HMaster.java:1071) > at > org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:863) > at > org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:612) > at java.lang.Thread.run(Thread.java:745) > > Exception in region server log > > 2014-09-24 20:10:16,535 WARN [RS_LOG_REPLAY_OPS-host-10-18-40-18:60020-1] > regionserver.SplitLogWorker: log splitting of > WALs/host-10-18-40-18,60020,1411558717849-splitting/host-10-18-40-18%2C60020%2C1411558717849.1411558724316.meta > failed, returning error > java.io.IOException: Cannot get log reader > at > org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:161) > at > org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:89) > at > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:660) > at > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:569) > at > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:282) > at > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:225) > at > org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:143) > at > org.apache.hadoop.hbase.regionserver.handler.HLogSplitterHandler.process(HLogSplitterHandler.java:82) > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.UnsupportedOperationException: Unable to find > suitable constructor for class > org.apache.hadoop.hbase.regionserver.wal.SecureWALCellCodec > at > org.apache.hadoop.hbase.util.ReflectionUtils.instantiateWithCustomCtor(ReflectionUtils.java:39) > at > org.apache.hadoop.hbase.regionserver.wal.WALCellCodec.create(WALCellCodec.java:101) > at > org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.getCodec(ProtobufLogReader.java:242) > at > org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initAfterCompression(ProtobufLogReader.java:247) > at > org.apache.hadoop.hbase.regionserver.wal.SecureProtobufLogReader.initAfterCompression(SecureProtobufLogReader.java:138) > at > org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:85) > at > org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:113) > ... 11 more > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.hbase.regionserver.wal.SecureWALCellCodec.(org.apache.hadoop.conf.Configuration, > org.apache.hadoop.hbase.regionserver.wal.CompressionContext) > at java.lang.Class.getConstructor0(Class.java:2849) > at java.lang.Class.getDeclaredConstructor(Class.java:2053) > at > org.apach
Master shuts down during log splitting on restart
Hi All. I am using 0.98.6 HBase. I observed that when I have the following value set in my hbase-site.xml file hbase.regionserver.wal.encryption false hbase.regionserver.hlog.reader.impl org.apache.hadoop.hbase.regionserver.wal.SecureProtobufLogReader hbase.regionserver.hlog.writer.impl org.apache.hadoop.hbase.regionserver.wal.SecureProtobufLogWriter And while log splitting on hbase service restart, master shutdown with following exception. Exception in master log 2014-09-24 17:14:28,590 FATAL [master:host-10-18-40-18:6] master.HMaster: Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.security.access.AccessController] 2014-09-24 17:14:28,590 FATAL [master:host-10-18-40-18:6] master.HMaster: Unhandled exception. Starting shutdown. java.io.IOException: error or interrupted while splitting logs in [hdfs://10.18.40.18:8020/tmp/hbase-ashish/hbase/WALs/host-10-18-40-18,60020,1411558717849-splitting] Task = installed = 6 done = 0 error = 6 at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:378) at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:415) at org.apache.hadoop.hbase.master.MasterFileSystem.splitMetaLog(MasterFileSystem.java:307) at org.apache.hadoop.hbase.master.MasterFileSystem.splitMetaLog(MasterFileSystem.java:298) at org.apache.hadoop.hbase.master.HMaster.splitMetaLogBeforeAssignment(HMaster.java:1071) at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:863) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:612) at java.lang.Thread.run(Thread.java:745) Exception in region server log 2014-09-24 20:10:16,535 WARN [RS_LOG_REPLAY_OPS-host-10-18-40-18:60020-1] regionserver.SplitLogWorker: log splitting of WALs/host-10-18-40-18,60020,1411558717849-splitting/host-10-18-40-18%2C60020%2C1411558717849.1411558724316.meta failed, returning error java.io.IOException: Cannot get log reader at org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:161) at org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:89) at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:660) at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:569) at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:282) at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:225) at org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:143) at org.apache.hadoop.hbase.regionserver.handler.HLogSplitterHandler.process(HLogSplitterHandler.java:82) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.UnsupportedOperationException: Unable to find suitable constructor for class org.apache.hadoop.hbase.regionserver.wal.SecureWALCellCodec at org.apache.hadoop.hbase.util.ReflectionUtils.instantiateWithCustomCtor(ReflectionUtils.java:39) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec.create(WALCellCodec.java:101) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.getCodec(ProtobufLogReader.java:242) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initAfterCompression(ProtobufLogReader.java:247) at org.apache.hadoop.hbase.regionserver.wal.SecureProtobufLogReader.initAfterCompression(SecureProtobufLogReader.java:138) at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:85) at org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:113) ... 11 more Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.hbase.regionserver.wal.SecureWALCellCodec.(org.apache.hadoop.conf.Configuration, org.apache.hadoop.hbase.regionserver.wal.CompressionContext) at java.lang.Class.getConstructor0(Class.java:2849) at java.lang.Class.getDeclaredConstructor(Class.java:2053) at org.apache.hadoop.hbase.util.ReflectionUtils.instantiateWithCustomCtor(ReflectionUtils.java:33) ... 17 more Question, When wal encryption is disabled should we set SecureWALCellCodec for cellCodecClsName in WALHeader.Builder object ? Regards, Ashish Singhi
