Furthermore, tservers being down shouldn't prevent the monitor page from coming up at all, so check for any other relevant messages in of the monitor's logs.
On Wed, Sep 11, 2013 at 2:08 PM, Pete Carlson <[email protected]> wrote: > How do I determine if the rest of Accumulo is up and running since I can't > use the monitor? > > Inside the monitor logs I see messages that it connected to Hadoop and > Zookeeper, but then later on I see this message stating there are no tablet > servers: > > 2013-09-11 11:39:29,199 [server.Accumulo] WARN : System swappiness > setting is greater than ten (60) which can cause time-sensitive operations > to be delayed. Accumulo is time sensitive because it needs to maintain > distributed lock agreement. > 2013-09-11 11:39:32,506 [impl.ServerClient] WARN : > tracer:localhost.localdomain There are no tablet servers: check that > zookeeper and accumulo are running. > > And when I log in to the accumulo shell I get the warning: There are no > tablet servers: check that zookeeper and accumulo are running. > > Could the system swappiness setting of 60 be causing my issue? > > Regards, > > Pete > > > On Wed, Sep 11, 2013 at 1:13 PM, John Vines <[email protected]> wrote: > >> Is the rest of Accumulo up and running? And is that the only type of >> error in the Monitor logs? >> >> >> On Wed, Sep 11, 2013 at 1:04 PM, Pete Carlson <[email protected]>wrote: >> >>> Ok, I deleted /accumulo out of hdfs, initialized accumulo, and >>> re-started Accumulo. I still can't see the monitor IN my VM browser. I >>> still get the same Monitor info message in the logs. >>> >>> Pete >>> >>> On Wed, Sep 11, 2013 at 11:55 AM, John Vines <[email protected]> wrote: >>> >>>> You need to delete the /accumulo directory out of hdfs >>>> >>>> >>>> On Wed, Sep 11, 2013 at 11:54 AM, Pete Carlson <[email protected]>wrote: >>>> >>>>> I removed the monitor certificates as suggested, and I no longer get >>>>> any messages in the monitor_localhost.err file. >>>>> >>>>> When I look in the monitor_localhost.localdomain.log file and saw the >>>>> following INFO/Exception message: >>>>> >>>>> 2013-09-11 11:42:50,012 [monitor.Monitor] INFO : Failed to obtain >>>>> problem reports >>>>> java.lang.RuntimeException: >>>>> org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException >>>>> at >>>>> org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:174) >>>>> at >>>>> org.apache.accumulo.server.problems.ProblemReports$3.hasNext(ProblemReports.java:241) >>>>> at >>>>> org.apache.accumulo.server.problems.ProblemReports.summarize(ProblemReports.java:299) >>>>> at >>>>> org.apache.accumulo.server.monitor.Monitor.fetchData(Monitor.java:392) >>>>> at >>>>> org.apache.accumulo.server.monitor.Monitor$2.run(Monitor.java:504) >>>>> at >>>>> org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34) >>>>> at java.lang.Thread.run(Thread.java:724) >>>>> Caused by: >>>>> org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException >>>>> at >>>>> org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:212) >>>>> at >>>>> org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:82) >>>>> at >>>>> org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:164) >>>>> ... 6 more >>>>> >>>>> I was thinking this was related to the fact I didn't yet have Thrift >>>>> 0.9.0 installed (due to dependency issues). I now have Thrift 0.9.0 >>>>> installed. I saved off my config files, blew away the accumulo directory >>>>> and extracted it again from the tar.gz and copied over my conf files. >>>>> >>>>> However, when I run the accumulo-1.5.0/bin/accumulo init command again >>>>> it gives me the following error: >>>>> >>>>> [util.Initialize] FATAL: It appears this location was previously >>>>> initialized, exiting ... >>>>> >>>>> I can still start Accumulo like before with the bin/start-all.sh >>>>> command but the Accumulo monitor still doesn't work IN my VM. >>>>> >>>>> Any other ideas to try? >>>>> >>>>> Pete >>>>> >>>>> On Wed, Sep 11, 2013 at 10:57 AM, John Vines <[email protected]> wrote: >>>>> >>>>>> Either try rerunning the monitor certificates or remove the portion >>>>>> from your accumulo-site that defines use of the certificate. I suggest >>>>>> the >>>>>> latter simply to remove variables at hand. >>>>>> >>>>>> >>>>>> On Wed, Sep 11, 2013 at 10:54 AM, Pete Carlson >>>>>> <[email protected]>wrote: >>>>>> >>>>>>> John, >>>>>>> >>>>>>> What is the monitor process called so I can verify its running? I >>>>>>> did look at the thread monitor logs, and under monitor_localhost.err it >>>>>>> gives a FileNotFoundException because it can't find the keystore.jks >>>>>>> under >>>>>>> accumulo-1.5.0/conf/ >>>>>>> >>>>>>> I would have thought when I ran ./generate_monitor_certificate.sh it >>>>>>> would have created this. >>>>>>> >>>>>>> I was trying to run the monitor in a browser IN the VM. Thats how I >>>>>>> found this problem. >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Pete >>>>>>> >>>>>>> >>>>>>> On Wed, Sep 11, 2013 at 10:41 AM, John Vines <[email protected]>wrote: >>>>>>> >>>>>>>> Again, please check to make sure the monitor process is running and >>>>>>>> it's not reporting any errors in it's logs. >>>>>>>> >>>>>>>> If you can verify that (and please verify that first), try >>>>>>>> accessing the monitor in a browser IN the VM. >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Sep 11, 2013 at 10:36 AM, Pete Carlson <[email protected] >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> All, >>>>>>>>> >>>>>>>>> Just to clarify I have a Centos VM running on my Windows 7 box, >>>>>>>>> and I have Accumulo, Hadoop, and Zookeeper running on it standalone. >>>>>>>>> I was >>>>>>>>> trying to open the Accumulo monitor within that VM and it failed. >>>>>>>>> However >>>>>>>>> I can open the Hadoop Namenode and Jobtracker web interfaces in my >>>>>>>>> browser >>>>>>>>> just fine. >>>>>>>>> >>>>>>>>> Brendan, I was able to connect to my VM using netcat on port 50095. >>>>>>>>> >>>>>>>>> However, when I try to telnet to that port on my Win7 with telnet >>>>>>>>> using my VM's ip address and port i.e., >>>>>>>>> >>>>>>>>> telnet <IP address of vm> 50095 >>>>>>>>> >>>>>>>>> I get a message: ....Could not open connection to the host, on >>>>>>>>> port 50095: Connect failed. >>>>>>>>> >>>>>>>>> Pete >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Sep 11, 2013 at 9:15 AM, Brendan Heussler < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Can you try to connect to your VM using netcat on port 50095? >>>>>>>>>> >>>>>>>>>> On VM: >>>>>>>>>> nc -l 50095 >>>>>>>>>> >>>>>>>>>> On Client: >>>>>>>>>> nc <ip-address of VM> 50095 >>>>>>>>>> >>>>>>>>>> Then type "test" or something on your Client machine. The >>>>>>>>>> message should be sent to your VM. >>>>>>>>>> >>>>>>>>>> This will help to test the connectivity of the two machines over >>>>>>>>>> port 50095 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Brendan >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Sep 10, 2013 at 5:22 PM, John Vines <[email protected]>wrote: >>>>>>>>>> >>>>>>>>>>> Are you certain the monitor is running and was able to bind to >>>>>>>>>>> the port? Please check the monitor logs for any errors. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Sep 10, 2013 at 5:18 PM, Pete Carlson < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> John, >>>>>>>>>>>> >>>>>>>>>>>> I don't see port 50095 in use in the VM. When I become root >>>>>>>>>>>> and type lsof -i -P | grep :50095 I don't get anything back. >>>>>>>>>>>> >>>>>>>>>>>> Any suggestions on how to get the port running? I tried >>>>>>>>>>>> editing the iptables with the following command: >>>>>>>>>>>> >>>>>>>>>>>> iptables -I INPUT -p tcp --dport 50095 -j ACCEPT >>>>>>>>>>>> >>>>>>>>>>>> and then I typed >>>>>>>>>>>> >>>>>>>>>>>> service iptables save >>>>>>>>>>>> >>>>>>>>>>>> to save my iptable changes, but when I restarted my VM I didn't >>>>>>>>>>>> see any difference. >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> >>>>>>>>>>>> Pete >>>>>>>>>>>> >>>>>>>>>>>> Looking for a unique woodcraft? Check out my Facebook page: >>>>>>>>>>>> www.facebook.com/oqwcrafts >>>>>>>>>>>> >>>>>>>>>>>> Also on Etsy: https://www.etsy.com/shop/OlneyQualityWoodCrft >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Sep 10, 2013 at 4:56 PM, John Vines >>>>>>>>>>>> <[email protected]>wrote: >>>>>>>>>>>> >>>>>>>>>>>>> You don't see 50095 in use locally on the VM or in your box? >>>>>>>>>>>>> You should only see it in use in the VM. If not, make sure it's >>>>>>>>>>>>> running. If >>>>>>>>>>>>> it is, check the monitor logs and see what's up. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Sep 10, 2013 at 4:51 PM, Pete Carlson < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi John, >>>>>>>>>>>>>> >>>>>>>>>>>>>> No, it looks like port 50095 wasn't open on my VM. I tried >>>>>>>>>>>>>> modifying the IP tables to make it opened, and restarted my VM >>>>>>>>>>>>>> but when I >>>>>>>>>>>>>> run the lsof -i command I don't see port 50095 listed. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Pete >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Pete >>>>>>>>>>>>>> >>>>>>>>>>>>>> Looking for a unique woodcraft? Check out my Facebook page: >>>>>>>>>>>>>> www.facebook.com/oqwcrafts >>>>>>>>>>>>>> >>>>>>>>>>>>>> Also on Etsy: https://www.etsy.com/shop/OlneyQualityWoodCrft >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, Sep 10, 2013 at 3:56 PM, John Vines <[email protected] >>>>>>>>>>>>>> > wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Do you have the port open to your VM? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Sep 10, 2013 at 3:54 PM, Pete Carlson < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I just set up a free standing Accumulo 1.5 instance on my >>>>>>>>>>>>>>>> linux VM, and when I run bin/start-all.sh it starts up >>>>>>>>>>>>>>>> accumulo, and also >>>>>>>>>>>>>>>> connects to my free standing Zookeeper and HDFS instances . >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> However, when I try to pull up the Accumulo master in my >>>>>>>>>>>>>>>> FireFox browser (Firefox ESR 10.0.12) i.e., >>>>>>>>>>>>>>>> http://accumulomaster:50095/status for health and >>>>>>>>>>>>>>>> monitoring, I get an unable to connect error. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I also tried running ./generate_monitor_certificate.sh and >>>>>>>>>>>>>>>> copied those properties in my accumulo-site.xml but I still >>>>>>>>>>>>>>>> get the same >>>>>>>>>>>>>>>> result. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Is there something else I need to configure? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Pete >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
