This reflection should occur only once, not at every write to the HLog, so the performance impact should be minimal, is it not?
why are you seeing the exception now? are u using a new unit test or a new hdfs jar? On Thu, Dec 1, 2011 at 9:59 PM, Mikhail Bautin < [email protected]> wrote: > Hello, > > The following reflection hack is from SequenceFileLogReader.java. > > try { > Field fIn = FilterInputStream.class.getDeclaredField("in"); > fIn.setAccessible(true); > Object realIn = fIn.get(this.in); > Method getFileLength = realIn.getClass(). > getMethod("getFileLength", new Class<?> []{}); > getFileLength.setAccessible(true); > long realLength = ((Long)getFileLength. > invoke(realIn, new Object []{})).longValue(); > assert(realLength >= this.length); > adjust = realLength - this.length; > } catch(Exception e) { > SequenceFileLogReader.LOG.warn( > "Error while trying to get accurate file length. " + > "Truncation / data loss may occur if RegionServers die.", e); > } > > In my testing this resulted in the following error: > > 11/12/01 21:40:07 WARN wal.SequenceFileLogReader: Error while trying to get > accurate file length. Truncation / data loss may occur if RegionServers > die. > java.lang.NoSuchMethodException: > > org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.getFileLength() > at java.lang.Class.getMethod(Class.java:1605) > at > > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader$WALReaderFSDataInputStream.getPos(SequenceFileLogReader.java:108) > at > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1434) > at > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1424) > at > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1419) > at > > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:57) > at > > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:158) > at > org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:681) > at > > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:821) > at > > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:734) > at > > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFileToTemp(HLogSplitter.java:382) > at > > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFileToTemp(HLogSplitter.java:351) > at > > org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:113) > at > > org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:266) > at > > org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:197) > at > > org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:165) > at java.lang.Thread.run(Thread.java:680) > > Besides, even when it works, the reflection-based solution is probably > _much_ slower than straightforward method access. Should we create a Hadoop > patch to expose the appropriate API call and get rid of the reflection > hack? > > Thanks, > --Mikhail > -- Subscribe to my posts at http://www.facebook.com/dhruba
