[
https://issues.apache.org/jira/browse/ACCUMULO-1998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844367#comment-13844367
]
John Vines commented on ACCUMULO-1998:
--------------------------------------
It is concerning that both +r and !0 have files in them. What is going on here?!
> Exception while attempting to bring up 1.6.0 TabletServer after moderate
> amount of work
> ---------------------------------------------------------------------------------------
>
> Key: ACCUMULO-1998
> URL: https://issues.apache.org/jira/browse/ACCUMULO-1998
> Project: Accumulo
> Issue Type: Bug
> Reporter: Michael Allen
> Priority: Critical
> Fix For: 1.6.0
>
>
> The reproduction steps around this are a little bit fuzzy but basically we
> ran a moderate workload against a 1.6.0 server. Encryption happened to be
> turned on but that doesn't seem to be germane to the problem. After doing a
> moderate amount of work, Accumulo is refusing to start up, spewing this error
> over and over to the log:
> {noformat}
> 2013-12-10 10:23:02,529 [tserver.TabletServer] WARN : exception while doing
> multi-scan
> java.lang.RuntimeException: java.io.IOException: Failed to open
> hdfs://10.10.1.115:9000/accumulo/tables/!0/table_info/A000042x.rf
> at
> org.apache.accumulo.tserver.TabletServer$ThriftClientHandler$LookupTask.run(TabletServer.java:1125)
> at
> org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at
> org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
> at
> org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: Failed to open
> hdfs://10.10.1.115:9000/accumulo/tables/!0/table_info/A000042x.rf
> at
> org.apache.accumulo.tserver.FileManager.reserveReaders(FileManager.java:333)
> at
> org.apache.accumulo.tserver.FileManager.access$500(FileManager.java:58)
> at
> org.apache.accumulo.tserver.FileManager$ScanFileManager.openFiles(FileManager.java:478)
> at
> org.apache.accumulo.tserver.FileManager$ScanFileManager.openFileRefs(FileManager.java:466)
> at
> org.apache.accumulo.tserver.FileManager$ScanFileManager.openFiles(FileManager.java:486)
> at
> org.apache.accumulo.tserver.Tablet$ScanDataSource.createIterator(Tablet.java:2027)
> at
> org.apache.accumulo.tserver.Tablet$ScanDataSource.iterator(Tablet.java:1989)
> at
> org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.seek(SourceSwitchingIterator.java:163)
> at org.apache.accumulo.tserver.Tablet.lookup(Tablet.java:1565)
> at org.apache.accumulo.tserver.Tablet.lookup(Tablet.java:1672)
> at
> org.apache.accumulo.tserver.TabletServer$ThriftClientHandler$LookupTask.run(TabletServer.java:1114)
> ... 6 more
> Caused by: java.io.FileNotFoundException: File does not exist:
> /accumulo/tables/!0/table_info/A000042x.rf
> at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.fetchLocatedBlocks(DFSClient.java:2006)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1975)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1967)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:735)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:165)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:436)
> at
> org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.getBCFile(CachableBlockFile.java:256)
> at
> org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.access$000(CachableBlockFile.java:143)
> at
> org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader$MetaBlockLoader.get(CachableBlockFile.java:212)
> at
> org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.getBlock(CachableBlockFile.java:313)
> at
> org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.getMetaBlock(CachableBlockFile.java:367)
> at
> org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.getMetaBlock(CachableBlockFile.java:143)
> at
> org.apache.accumulo.core.file.rfile.RFile$Reader.<init>(RFile.java:825)
> at
> org.apache.accumulo.core.file.rfile.RFileOperations.openReader(RFileOperations.java:79)
> at
> org.apache.accumulo.core.file.DispatchingFileFactory.openReader(FileOperations.java:119)
> at
> org.apache.accumulo.tserver.FileManager.reserveReaders(FileManager.java:314)
> ... 16 more
> {noformat}
> Here's some other pieces of context:
> HDFS contents:
> {noformat}
> ubuntu@ip-10-10-1-115:/data0/logs/accumulo$ hadoop fs -lsr /accumulo/tables/
> drwxr-xr-x - accumulo hadoop 0 2013-12-10 00:32 /accumulo/tables/!0
> drwxr-xr-x - accumulo hadoop 0 2013-12-10 01:06
> /accumulo/tables/!0/default_tablet
> drwxr-xr-x - accumulo hadoop 0 2013-12-10 10:49
> /accumulo/tables/!0/table_info
> -rw-r--r-- 5 accumulo hadoop 1698 2013-12-10 00:34
> /accumulo/tables/!0/table_info/F0000000.rf
> -rw-r--r-- 5 accumulo hadoop 43524 2013-12-10 01:53
> /accumulo/tables/!0/table_info/F000062q.rf
> drwxr-xr-x - accumulo hadoop 0 2013-12-10 00:32 /accumulo/tables/+r
> drwxr-xr-x - accumulo hadoop 0 2013-12-10 10:45
> /accumulo/tables/+r/root_tablet
> -rw-r--r-- 5 accumulo hadoop 2070 2013-12-10 10:45
> /accumulo/tables/+r/root_tablet/A0000738.rf
> drwxr-xr-x - accumulo hadoop 0 2013-12-10 00:33 /accumulo/tables/1
> drwxr-xr-x - accumulo hadoop 0 2013-12-10 00:33
> /accumulo/tables/1/default_tablet
> {noformat}
> ZooKeeper entries
> {noformat}
> [zk: localhost:2181(CONNECTED) 6] get
> /accumulo/371cfa3e-fe96-4a50-92e9-da7572589ffa/root_tablet/dir
> hdfs://10.10.1.115:9000/accumulo/tables/+r/root_tablet
> cZxid = 0x1b
> ctime = Tue Dec 10 00:32:56 EST 2013
> mZxid = 0x1b
> mtime = Tue Dec 10 00:32:56 EST 2013
> pZxid = 0x1b
> cversion = 0
> dataVersion = 0
> aclVersion = 0
> ephemeralOwner = 0x0
> dataLength = 54
> numChildren = 0
> {noformat}
> I'm going to preserve the state of this machine in HDFS for a while but not
> forever, so if there are other pieces of context people need, let me know.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
