Hi,
I noticed that in the monitor_nodejs.whatever.com.debug.log file I am seeing
these messages:
2017-05-23 01:56:51,035 [impl.ThriftScanner] DEBUG: Failed to locate tablet
for table : !0 row : ~err_
2017-05-23 01:56:57,173 [monitor.Monitor] INFO : Failed to obtain problem
reports
java.lang.RuntimeException:
org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException
at
org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:160)
at
org.apache.accumulo.server.problems.ProblemReports$3.hasNext(ProblemReports.java:252)
at
org.apache.accumulo.server.problems.ProblemReports.summarize(ProblemReports.java:310)
at org.apache.accumulo.monitor.Monitor.fetchData(Monitor.java:374)
at org.apache.accumulo.monitor.Monitor$2.run(Monitor.java:519)
at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
at java.lang.Thread.run(Thread.java:745)
Caused by:
org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException
at
org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:243)
at
org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:79)
at
org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:150)
... 6 more
2017-05-23 01:56:57,512 [impl.ThriftScanner] DEBUG: Failed to locate tablet
for table : !0 row : ~err_
2017-05-23 01:57:03,604 [monitor.Monitor] INFO : Failed to obtain problem
reports
java.lang.RuntimeException:
org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException
at
org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:160)
at
org.apache.accumulo.server.problems.ProblemReports$3.hasNext(ProblemReports.java:252)
at
org.apache.accumulo.server.problems.ProblemReports.summarize(ProblemReports.java:310)
at org.apache.accumulo.monitor.Monitor.fetchData(Monitor.java:374)
at org.apache.accumulo.monitor.Monitor$2.run(Monitor.java:519)
at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
at java.lang.Thread.run(Thread.java:745)
Caused by:
org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException
at
org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:243)
at
org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:79)
at
org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:150)
... 6 more
2017-05-23 01:57:03,944 [impl.ThriftScanner] DEBUG: Failed to locate tablet
for table : !0 row : ~err_
--------------------------------------------
On Tue, 5/23/17, o haya <[email protected]> wrote:
Subject: scan of small table stopped working (hangs)
To: [email protected]
Date: Tuesday, May 23, 2017, 1:56 AM
Hi,
I have a small Accumulo test instance,
that I installed following this page:
https://www.digitalocean.com/community/tutorials/how-to-install-the-big-data-friendly-apache-accumulo-nosql-database-on-ubuntu-14-04
I've been doing testing and was able to
successfully run the SimpleIngestClient from the minicluster
project and create a table, testTable.
After that, I could use the Accumulo
shell and scan the table, after logging in as a user, "demo"
and running "scan -t testTable".
However, tonight, after I brought up
the machine and started Hadoop
(/apps/hadoop/sbin/start-dfs.sh), then zookeeper
(/apps/zookeeper/bin/zkServer.sh start) and finally,
Accumulo (/apps/accumulo/bin/start-all.sh).
And then I started the Accumulo shell
and logged in as "demo" and did "scan -t testTable" and it
just hangs, with no output.
I found another thread that suggested
enabling "debug on" and here is the output:
root@accumulo1>
debug on
2017-05-23 01:44:37,524
[admin.TableOperations] TRACE: tid=13 Fetching list of
tables...
2017-05-23 01:44:37,526
[admin.TableOperations] TRACE: tid=13 Fetched 5 table names
in 0.001 secs
2017-05-23 01:44:37,526
[rpc.ThriftUtil] TRACE: Opening normal transport
2017-05-23 01:44:37,532
[impl.ThriftTransportPool] TRACE: Creating new connection to
connection to localhost:9997
2017-05-23 01:44:37,534
[impl.ThriftTransportPool] TRACE: Returned connection
localhost:9997 (120000) ioCount: 170
2017-05-23 01:44:37,534
[admin.TableOperations] TRACE: tid=13 Fetching list of
namespaces...
2017-05-23 01:44:37,534
[admin.TableOperations] TRACE: tid=13 Fetched 2 namespaces
in 0.000 secs
root@accumulo1>
scan -t testTable
2017-05-23 01:44:51,547
[admin.TableOperations] TRACE: tid=13 Checking if table
testTable exists...
2017-05-23 01:44:51,547
[admin.TableOperations] TRACE: tid=13 Checked existance of
true in 0.000 secs
2017-05-23 01:44:51,548
[rpc.ThriftUtil] TRACE: Opening normal transport
2017-05-23 01:44:51,549
[impl.ThriftTransportPool] TRACE: Creating new connection to
connection to localhost:9997
2017-05-23 01:44:51,565
[impl.ThriftTransportPool] TRACE: Returned connection
localhost:9997 (120000) ioCount: 6808
2017-05-23 01:44:51,565
[admin.TableOperations] TRACE: tid=13 Checking if table
testTable exists...
2017-05-23 01:44:51,566
[admin.TableOperations] TRACE: tid=13 Checked existance of
true in 0.000 secs
2017-05-23 01:44:51,566
[impl.ThriftTransportPool] TRACE: Using existing connection
to localhost:9997
2017-05-23 01:44:51,582
[impl.ThriftTransportPool] TRACE: Returned connection
localhost:9997 (120000) ioCount: 6808
2017-05-23 01:44:51,583
[impl.ThriftTransportPool] TRACE: Using existing connection
to localhost:9997
2017-05-23 01:44:51,594
[impl.ThriftTransportPool] TRACE: Returned connection
localhost:9997 (120000) ioCount: 6808
2017-05-23 01:44:51,594
[admin.TableOperations] TRACE: tid=13 Checking if table
testTable exists...
2017-05-23 01:44:51,594
[admin.TableOperations] TRACE: tid=13 Checked existance of
true in 0.000 secs
2017-05-23 01:44:51,594
[impl.ThriftTransportPool] TRACE: Using existing connection
to localhost:9997
2017-05-23 01:44:51,608
[impl.ThriftTransportPool] TRACE: Returned connection
localhost:9997 (120000) ioCount: 6808
2017-05-23 01:44:51,609
[impl.ThriftTransportPool] TRACE: Using existing connection
to localhost:9997
2017-05-23 01:44:51,614
[impl.ThriftTransportPool] TRACE: Returned connection
localhost:9997 (120000) ioCount: 130
2017-05-23 01:44:51,620 [shell.Shell]
DEBUG: Found no scan iterators to set
2017-05-23 01:44:51,644
[impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
table=7 row= skipRow=false retry=false
2017-05-23 01:44:51,644
[impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
table=!0 row=7; skipRow=false retry=false
2017-05-23 01:44:51,644
[impl.RootTabletLocator] TRACE: tid=13 Looking up root
tablet location in zookeeper.
2017-05-23 01:44:51,658
[impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
null in 0.014 secs
2017-05-23 01:44:51,659
[impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
at null in 0.014 secs
2017-05-23 01:44:51,659
[impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
at null in 0.015 secs
2017-05-23 01:44:51,659
[impl.ThriftScanner] DEBUG: Failed to locate tablet for
table : 7 row :
2017-05-23 01:44:51,759
[impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
table=7 row= skipRow=false retry=false
2017-05-23 01:44:51,759
[impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
table=!0 row=7; skipRow=false retry=false
2017-05-23 01:44:51,759
[impl.RootTabletLocator] TRACE: tid=13 Looking up root
tablet location in zookeeper.
2017-05-23 01:44:51,759
[impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
null in 0.000 secs
2017-05-23 01:44:51,759
[impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
at null in 0.000 secs
2017-05-23 01:44:51,760
[impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
at null in 0.000 secs
2017-05-23 01:44:51,760
[impl.ThriftScanner] TRACE: Failed to locate tablet for
table : 7 row :
2017-05-23 01:44:51,952
[impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
table=7 row= skipRow=false retry=false
2017-05-23 01:44:51,952
[impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
table=!0 row=7; skipRow=false retry=false
2017-05-23 01:44:51,952
[impl.RootTabletLocator] TRACE: tid=13 Looking up root
tablet location in zookeeper.
2017-05-23 01:44:51,952
[impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
null in 0.000 secs
2017-05-23 01:44:51,952
[impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
at null in 0.000 secs
2017-05-23 01:44:51,953
[impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
at null in 0.000 secs
2017-05-23 01:44:51,953
[impl.ThriftScanner] TRACE: Failed to locate tablet for
table : 7 row :
2017-05-23 01:44:52,302
[impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
table=7 row= skipRow=false retry=false
2017-05-23 01:44:52,302
[impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
table=!0 row=7; skipRow=false retry=false
2017-05-23 01:44:52,302
[impl.RootTabletLocator] TRACE: tid=13 Looking up root
tablet location in zookeeper.
2017-05-23 01:44:52,303
[impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
null in 0.000 secs
2017-05-23 01:44:52,303
[impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
at null in 0.000 secs
2017-05-23 01:44:52,303
[impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
at null in 0.001 secs
2017-05-23 01:44:52,303
[impl.ThriftScanner] TRACE: Failed to locate tablet for
table : 7 row :
2017-05-23 01:44:52,960
[impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
table=7 row= skipRow=false retry=false
2017-05-23 01:44:52,960
[impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
table=!0 row=7; skipRow=false retry=false
2017-05-23 01:44:52,960
[impl.RootTabletLocator] TRACE: tid=13 Looking up root
tablet location in zookeeper.
2017-05-23 01:44:52,960
[impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
null in 0.000 secs
2017-05-23 01:44:52,960
[impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
at null in 0.000 secs
2017-05-23 01:44:52,960
[impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
at null in 0.000 secs
2017-05-23 01:44:52,960
[impl.ThriftScanner] TRACE: Failed to locate tablet for
table : 7 row :
2017-05-23 01:44:54,282
[impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
table=7 row= skipRow=false retry=false
2017-05-23 01:44:54,282
[impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
table=!0 row=7; skipRow=false retry=false
2017-05-23 01:44:54,282
[impl.RootTabletLocator] TRACE: tid=13 Looking up root
tablet location in zookeeper.
2017-05-23 01:44:54,282
[impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
null in 0.000 secs
2017-05-23 01:44:54,282
[impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
at null in 0.000 secs
2017-05-23 01:44:54,282
[impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
at null in 0.000 secs
2017-05-23 01:44:54,282
[impl.ThriftScanner] TRACE: Failed to locate tablet for
table : 7 row :
2017-05-23 01:44:56,898
[impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
table=7 row= skipRow=false retry=false
2017-05-23 01:44:56,898
[impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
table=!0 row=7; skipRow=false retry=false
2017-05-23 01:44:56,898
[impl.RootTabletLocator] TRACE: tid=13 Looking up root
tablet location in zookeeper.
2017-05-23 01:44:56,898
[impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
null in 0.000 secs
2017-05-23 01:44:56,898
[impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
at null in 0.000 secs
2017-05-23 01:44:56,898
[impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
at null in 0.000 secs
2017-05-23 01:44:56,898
[impl.ThriftScanner] TRACE: Failed to locate tablet for
table : 7 row :
2017-05-23 01:45:02,204
[impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
table=7 row= skipRow=false retry=false
2017-05-23 01:45:02,204
[impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
table=!0 row=7; skipRow=false retry=false
2017-05-23 01:45:02,204
[impl.RootTabletLocator] TRACE: tid=13 Looking up root
tablet location in zookeeper.
2017-05-23 01:45:02,204
[impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
null in 0.000 secs
2017-05-23 01:45:02,204
[impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
at null in 0.000 secs
2017-05-23 01:45:02,205
[impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
at null in 0.000 secs
2017-05-23 01:45:02,205
[impl.ThriftScanner] TRACE: Failed to locate tablet for
table : 7 row :
2017-05-23 01:45:07,560
[impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
table=7 row= skipRow=false retry=false
2017-05-23 01:45:07,560
[impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
table=!0 row=7; skipRow=false retry=false
2017-05-23 01:45:07,560
[impl.RootTabletLocator] TRACE: tid=13 Looking up root
tablet location in zookeeper.
2017-05-23 01:45:07,560
[impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
null in 0.000 secs
2017-05-23 01:45:07,561
[impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
at null in 0.000 secs
2017-05-23 01:45:07,561
[impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
at null in 0.000 secs
2017-05-23 01:45:07,561
[impl.ThriftScanner] TRACE: Failed to locate tablet for
table : 7 row :
2017-05-23 01:45:13,002
[impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
table=7 row= skipRow=false retry=false
2017-05-23 01:45:13,002
[impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
table=!0 row=7; skipRow=false retry=false
2017-05-23 01:45:13,002
[impl.RootTabletLocator] TRACE: tid=13 Looking up root
tablet location in zookeeper.
2017-05-23 01:45:13,002
[impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
null in 0.000 secs
2017-05-23 01:45:13,002
[impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
at null in 0.000 secs
2017-05-23 01:45:13,002
[impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
at null in 0.000 secs
2017-05-23 01:45:13,002
[impl.ThriftScanner] TRACE: Failed to locate tablet for
table : 7 row :
2017-05-23 01:45:17,530
[impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
table=7 row= skipRow=false retry=false
2017-05-23 01:45:17,530
[impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
table=!0 row=7; skipRow=false retry=false
2017-05-23 01:45:17,530
[impl.RootTabletLocator] TRACE: tid=13 Looking up root
tablet location in zookeeper.
2017-05-23 01:45:17,530
[impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
null in 0.000 secs
2017-05-23 01:45:17,530
[impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
at null in 0.000 secs
2017-05-23 01:45:17,530
[impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
at null in 0.000 secs
2017-05-23 01:45:17,530
[impl.ThriftScanner] TRACE: Failed to locate tablet for
table : 7 row :
2017-05-23 01:45:22,802
[impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
table=7 row= skipRow=false retry=false
2017-05-23 01:45:22,803
[impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
table=!0 row=7; skipRow=false retry=false
2017-05-23 01:45:22,803
[impl.RootTabletLocator] TRACE: tid=13 Looking up root
tablet location in zookeeper.
2017-05-23 01:45:22,803
[impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
null in 0.000 secs
2017-05-23 01:45:22,803
[impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
at null in 0.000 secs
2017-05-23 01:45:22,803
[impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
at null in 0.000 secs
2017-05-23 01:45:22,803
[impl.ThriftScanner] TRACE: Failed to locate tablet for
table : 7 row :
2017-05-23 01:45:28,247
[impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
table=7 row= skipRow=false retry=false
2017-05-23 01:45:28,247
[impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
table=!0 row=7; skipRow=false retry=false
2017-05-23 01:45:28,247
[impl.RootTabletLocator] TRACE: tid=13 Looking up root
tablet location in zookeeper.
2017-05-23 01:45:28,248
[impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
null in 0.000 secs
2017-05-23 01:45:28,248
[impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
at null in 0.000 secs
2017-05-23 01:45:28,248
[impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
at null in 0.000 secs
2017-05-23 01:45:28,248
[impl.ThriftScanner] TRACE: Failed to locate tablet for
table : 7 row :
2017-05-23 01:45:30,781 [ipc.Client]
DEBUG: stopping client from cache: org.apache.hadoop.ipc.Client@a95842c
2017-05-23 01:45:30,782 [ipc.Client]
DEBUG: removing client from cache: org.apache.hadoop.ipc.Client@a95842c
2017-05-23 01:45:30,782 [ipc.Client]
DEBUG: stopping actual client because no more references
remain: org.apache.hadoop.ipc.Client@a95842c
2017-05-23 01:45:30,782 [ipc.Client]
DEBUG: Stopping client
Can anyone tell from the above why the
scan command is no longer working?
Thanks,
Jim
