Check the tserver logs.
On Wed, Sep 11, 2013 at 2:08 PM, Pete Carlson <[email protected]> wrote: > How do I determine if the rest of Accumulo is up and running since I can't > use the monitor? > > Inside the monitor logs I see messages that it connected to Hadoop and > Zookeeper, but then later on I see this message stating there are no tablet > servers: > > 2013-09-11 11:39:29,199 [server.Accumulo] WARN : System swappiness > setting is greater than ten (60) which can cause time-sensitive operations > to be delayed. Accumulo is time sensitive because it needs to maintain > distributed lock agreement. > 2013-09-11 11:39:32,506 [impl.ServerClient] WARN : > tracer:localhost.localdomain There are no tablet servers: check that > zookeeper and accumulo are running. > > And when I log in to the accumulo shell I get the warning: There are no > tablet servers: check that zookeeper and accumulo are running. > > Could the system swappiness setting of 60 be causing my issue? > > Regards, > > Pete > > > On Wed, Sep 11, 2013 at 1:13 PM, John Vines <[email protected]> wrote: > >> Is the rest of Accumulo up and running? And is that the only type of >> error in the Monitor logs? >> >> >> On Wed, Sep 11, 2013 at 1:04 PM, Pete Carlson <[email protected]>wrote: >> >>> Ok, I deleted /accumulo out of hdfs, initialized accumulo, and >>> re-started Accumulo. I still can't see the monitor IN my VM browser. I >>> still get the same Monitor info message in the logs. >>> >>> Pete >>> >>> On Wed, Sep 11, 2013 at 11:55 AM, John Vines <[email protected]> wrote: >>> >>>> You need to delete the /accumulo directory out of hdfs >>>> >>>> >>>> On Wed, Sep 11, 2013 at 11:54 AM, Pete Carlson <[email protected]>wrote: >>>> >>>>> I removed the monitor certificates as suggested, and I no longer get >>>>> any messages in the monitor_localhost.err file. >>>>> >>>>> When I look in the monitor_localhost.localdomain.log file and saw the >>>>> following INFO/Exception message: >>>>> >>>>> 2013-09-11 11:42:50,012 [monitor.Monitor] INFO : Failed to obtain >>>>> problem reports >>>>> java.lang.RuntimeException: >>>>> org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException >>>>> at >>>>> org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:174) >>>>> at >>>>> org.apache.accumulo.server.problems.ProblemReports$3.hasNext(ProblemReports.java:241) >>>>> at >>>>> org.apache.accumulo.server.problems.ProblemReports.summarize(ProblemReports.java:299) >>>>> at >>>>> org.apache.accumulo.server.monitor.Monitor.fetchData(Monitor.java:392) >>>>> at >>>>> org.apache.accumulo.server.monitor.Monitor$2.run(Monitor.java:504) >>>>> at >>>>> org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34) >>>>> at java.lang.Thread.run(Thread.java:724) >>>>> Caused by: >>>>> org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException >>>>> at >>>>> org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:212) >>>>> at >>>>> org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:82) >>>>> at >>>>> org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:164) >>>>> ... 6 more >>>>> >>>>> I was thinking this was related to the fact I didn't yet have Thrift >>>>> 0.9.0 installed (due to dependency issues). I now have Thrift 0.9.0 >>>>> installed. I saved off my config files, blew away the accumulo directory >>>>> and extracted it again from the tar.gz and copied over my conf files. >>>>> >>>>> However, when I run the accumulo-1.5.0/bin/accumulo init command again >>>>> it gives me the following error: >>>>> >>>>> [util.Initialize] FATAL: It appears this location was previously >>>>> initialized, exiting ... >>>>> >>>>> I can still start Accumulo like before with the bin/start-all.sh >>>>> command but the Accumulo monitor still doesn't work IN my VM. >>>>> >>>>> Any other ideas to try? >>>>> >>>>> Pete >>>>> >>>>> On Wed, Sep 11, 2013 at 10:57 AM, John Vines <[email protected]> wrote: >>>>> >>>>>> Either try rerunning the monitor certificates or remove the portion >>>>>> from your accumulo-site that defines use of the certificate. I suggest >>>>>> the >>>>>> latter simply to remove variables at hand. >>>>>> >>>>>> >>>>>> On Wed, Sep 11, 2013 at 10:54 AM, Pete Carlson >>>>>> <[email protected]>wrote: >>>>>> >>>>>>> John, >>>>>>> >>>>>>> What is the monitor process called so I can verify its running? I >>>>>>> did look at the thread monitor logs, and under monitor_localhost.err it >>>>>>> gives a FileNotFoundException because it can't find the keystore.jks >>>>>>> under >>>>>>> accumulo-1.5.0/conf/ >>>>>>> >>>>>>> I would have thought when I ran ./generate_monitor_certificate.sh it >>>>>>> would have created this. >>>>>>> >>>>>>> I was trying to run the monitor in a browser IN the VM. Thats how I >>>>>>> found this problem. >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Pete >>>>>>> >>>>>>> >>>>>>> On Wed, Sep 11, 2013 at 10:41 AM, John Vines <[email protected]>wrote: >>>>>>> >>>>>>>> Again, please check to make sure the monitor process is running and >>>>>>>> it's not reporting any errors in it's logs. >>>>>>>> >>>>>>>> If you can verify that (and please verify that first), try >>>>>>>> accessing the monitor in a browser IN the VM. >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Sep 11, 2013 at 10:36 AM, Pete Carlson <[email protected] >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> All, >>>>>>>>> >>>>>>>>> Just to clarify I have a Centos VM running on my Windows 7 box, >>>>>>>>> and I have Accumulo, Hadoop, and Zookeeper running on it standalone. >>>>>>>>> I was >>>>>>>>> trying to open the Accumulo monitor within that VM and it failed. >>>>>>>>> However >>>>>>>>> I can open the Hadoop Namenode and Jobtracker web interfaces in my >>>>>>>>> browser >>>>>>>>> just fine. >>>>>>>>> >>>>>>>>> Brendan, I was able to connect to my VM using netcat on port 50095. >>>>>>>>> >>>>>>>>> However, when I try to telnet to that port on my Win7 with telnet >>>>>>>>> using my VM's ip address and port i.e., >>>>>>>>> >>>>>>>>> telnet <IP address of vm> 50095 >>>>>>>>> >>>>>>>>> I get a message: ....Could not open connection to the host, on >>>>>>>>> port 50095: Connect failed. >>>>>>>>> >>>>>>>>> Pete >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Sep 11, 2013 at 9:15 AM, Brendan Heussler < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Can you try to connect to your VM using netcat on port 50095? >>>>>>>>>> >>>>>>>>>> On VM: >>>>>>>>>> nc -l 50095 >>>>>>>>>> >>>>>>>>>> On Client: >>>>>>>>>> nc <ip-address of VM> 50095 >>>>>>>>>> >>>>>>>>>> Then type "test" or something on your Client machine. The >>>>>>>>>> message should be sent to your VM. >>>>>>>>>> >>>>>>>>>> This will help to test the connectivity of the two machines over >>>>>>>>>> port 50095 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Brendan >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Sep 10, 2013 at 5:22 PM, John Vines <[email protected]>wrote: >>>>>>>>>> >>>>>>>>>>> Are you certain the monitor is running and was able to bind to >>>>>>>>>>> the port? Please check the monitor logs for any errors. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Sep 10, 2013 at 5:18 PM, Pete Carlson < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> John, >>>>>>>>>>>> >>>>>>>>>>>> I don't see port 50095 in use in the VM. When I become root >>>>>>>>>>>> and type lsof -i -P | grep :50095 I don't get anything back. >>>>>>>>>>>> >>>>>>>>>>>> Any suggestions on how to get the port running? I tried >>>>>>>>>>>> editing the iptables with the following command: >>>>>>>>>>>> >>>>>>>>>>>> iptables -I INPUT -p tcp --dport 50095 -j ACCEPT >>>>>>>>>>>> >>>>>>>>>>>> and then I typed >>>>>>>>>>>> >>>>>>>>>>>> service iptables save >>>>>>>>>>>> >>>>>>>>>>>> to save my iptable changes, but when I restarted my VM I didn't >>>>>>>>>>>> see any difference. >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> >>>>>>>>>>>> Pete >>>>>>>>>>>> >>>>>>>>>>>> Looking for a unique woodcraft? Check out my Facebook page: >>>>>>>>>>>> www.facebook.com/oqwcrafts >>>>>>>>>>>> >>>>>>>>>>>> Also on Etsy: https://www.etsy.com/shop/OlneyQualityWoodCrft >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Sep 10, 2013 at 4:56 PM, John Vines >>>>>>>>>>>> <[email protected]>wrote: >>>>>>>>>>>> >>>>>>>>>>>>> You don't see 50095 in use locally on the VM or in your box? >>>>>>>>>>>>> You should only see it in use in the VM. If not, make sure it's >>>>>>>>>>>>> running. If >>>>>>>>>>>>> it is, check the monitor logs and see what's up. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Sep 10, 2013 at 4:51 PM, Pete Carlson < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi John, >>>>>>>>>>>>>> >>>>>>>>>>>>>> No, it looks like port 50095 wasn't open on my VM. I tried >>>>>>>>>>>>>> modifying the IP tables to make it opened, and restarted my VM >>>>>>>>>>>>>> but when I >>>>>>>>>>>>>> run the lsof -i command I don't see port 50095 listed. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Pete >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Pete >>>>>>>>>>>>>> >>>>>>>>>>>>>> Looking for a unique woodcraft? Check out my Facebook page: >>>>>>>>>>>>>> www.facebook.com/oqwcrafts >>>>>>>>>>>>>> >>>>>>>>>>>>>> Also on Etsy: https://www.etsy.com/shop/OlneyQualityWoodCrft >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, Sep 10, 2013 at 3:56 PM, John Vines <[email protected] >>>>>>>>>>>>>> > wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Do you have the port open to your VM? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Sep 10, 2013 at 3:54 PM, Pete Carlson < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I just set up a free standing Accumulo 1.5 instance on my >>>>>>>>>>>>>>>> linux VM, and when I run bin/start-all.sh it starts up >>>>>>>>>>>>>>>> accumulo, and also >>>>>>>>>>>>>>>> connects to my free standing Zookeeper and HDFS instances . >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> However, when I try to pull up the Accumulo master in my >>>>>>>>>>>>>>>> FireFox browser (Firefox ESR 10.0.12) i.e., >>>>>>>>>>>>>>>> http://accumulomaster:50095/status for health and >>>>>>>>>>>>>>>> monitoring, I get an unable to connect error. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I also tried running ./generate_monitor_certificate.sh and >>>>>>>>>>>>>>>> copied those properties in my accumulo-site.xml but I still >>>>>>>>>>>>>>>> get the same >>>>>>>>>>>>>>>> result. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Is there something else I need to configure? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Pete >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
